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ABSTRACT 



This thesis reviewed 45 studies that purported to 
empirically assess the effectiveness of information systems. 
They were evaluated for (a) vulnerability to alternative 
explanations of reported findings, (b) effect of data 
collection methods upon their conclusions, (c) capacity to 
establish causal inference, (d) how they defined system 
success and (e) reported changes in organizations following 
information system implementation. 

MIS research methods do not provide a basis for 
establishing causal inference. This, in turn, denies the 
possibility of constructing a formal theory to describe, 
explain or predict the successful implementation of 
information systems. This deficiency is worsened by the 
emergence of antagonistic schools of thought among MIS 
researchers as to how system effectiveness should be 
evaluated . 

Recommendations are made for improvements in MIS research 
designs and data collection techniques. Suggestions are made 
for ways by which line managers and commanders as well as 
managers of MIS activities must become informed consumers of 
MIS academic research on system effectiveness. 
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INTRODUCTION 



I . 



The point of implementing computer-based information 
systems is to gain either increased operational efficiency or 
enhanced managerial effectiveness. These gains, however, are 
more often asserted or assumed than they are demonstrated or 
tested (Lotus, 1988/ Fawcette, 1988/ Bulkeley, 1987). 

Starting in 1985, senior line managers and information 
system managers in the United States began to ask whether 
they were getting a payoff from their investment in 
information systems, especially since the advent of the 
personal computer. 

A similar skepticism could well be evident in other 
countries (Maginnis, 1986/Bowen, 1986/ Strassmann, 1985/ 
Wessel, 1988/ LaPlante, 1988). 

A. GROWING DOUBT ABOUT THE PAYOFF FROM INFORMATION SYSTEMS 

Steve Crummey (1988), of Lotus Development Corporation, 
assessed the current information system productivity dilemma, 
at least as it applied to the proliferation of desk top 
microcomputers: "In 1976, at the dawn of the personal 
computer age, corporations invested $100 million in PC 
(personal computer) technology. Ten years later in 1986, they 
spent $46 billion on PCs — and nobody really knows what those 
dollars are buying." 
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1 . Questions That Arise 

Seeds of doubt about the assumption that the 
automation of management information systems (MIS) leads to 
increased productivity provoke questions about the ways in 
which the success of these systems are evaluated: 

a. What is meant by a payoff, success or benefit from the 
implementation of an information system? 

b. What are the magnitudes of improvement in organizational 
functioning reported by formal studies of information 
system implementation? 

c. What research tools are employed to gather data on the 
success or failure of information systems? 

d. To what extent do the findings reported by formal studies 
of system effectiveness have scientific credibility? 

2 . Answering Those Questions 

To address these questions, this thesis reviewed 45 
formal studies, ostensibly dealing with the measurement of 
the effectiveness of information systems. They were examined 
for : 

a. How they defined effectiveness, benefit or success from 
a system implementation. 

b. How their research designs dealt with threats to 
validity . 

c. The means by which they gathered data on system 
effectiveness . 

d. Reported levels of improvement in organizational 
functioning following the implementation of a system. 

The findings of this review are analyzed for their 
implications for: 

- Senior line managers and commanders as well as managers 
of information system function. 

- The intellectual foundations of the study of management 
information systems. 
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B. THE NEED TO STUDY PAYOFFS FROM INFORMATION SYSTEMS 

After spending billions of dollars on automating the 
offices of white collar workers, organizations find that they 
must now scrutinize the claims by computer vendors about 
increased productivity. To effectively manage a resource an 
organization must have a means to measure its benefits as well 
as its costs. 

An appropriate managerial question that arises is whether 
an organization is getting a return on its investment in a 
resource. This necessity is not diminished because the 
resource in question is a management information system. 
Managers of information systems are pressed by senior line 
management to answer, again and again, the question, "What is 
the value of the company's investment in information systems?" 
(Matlin, 1979) . 

According to Strassmann (1976), "Industries and 
governments all over the world are currently struggling to 
contain rising administrative and clerical overheads by 
automating information handling in the office." 

While the clerical work force has been the fastest growing 
segment of the labor force in nearly every industrialized 
country since the 1960s, the productivity of this sector has 
been flat since the early 1970s (Harris, 1987). This has been 
the result despite the fact that 40% of capital spending (in 
the U.S.) has been directed at information technology. 

General managers in the United States surveyed by the 
University of Minnesota Management information Systems 
Research Center (Brancheau and Wetherbe, 1987) ranked the 
measuring of system effectiveness as the fourth most critical 
problem they face in managing information systems. Sixty per 
cent of the Management Information Systems executives polled 
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in the same survey placed the measuring of effectiveness in 
their Top Ten list of critical problems. The assessment of 
system success has been in the Top Ten list of critical MIS 
problems since it was first formulated in 1980. 

C . THE HEED TO DEVELOP A FORMAL THEORY OF SYSTEM 
EFFECTIVENESS 

Clearly, then, the evaluation of the success of 
information systems is viewed by senior line managers, and by 
MIS managers, as being in the forefront of their concerns. 
While these executives represent the sectors who have the most 
at stake in the successful implementation and exploitation of 
information systems, in the social division of labor they are 
not the agents of measurement. 

That task falls to a neutral sector of the information 
systems realm: the academic scholars of MIS. Management must 
examine not only the effectiveness of information systems but 
also the effectiveness of the ways by which effectiveness 
itself is studied. It is the job of the MIS scholar to 
discover and systematize what is known about system 
effectiveness. A theory of system implementation would express 
what is known, what is predicted and what is assumed about the 
success of information systems. Without a formal theory of 
system implementation, we cannot competently describe, explain 
or predict. 

1. Functions of a Theory of System Effectiveness 

A formal theory of system effectiveness would be the 
basis for describing, explaining and predicting system 
effectiveness (Denzin, 1970) : 

• Description --Formal theory permits the description of 
a phenomenon is such a way that others can repeat a 
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study with a high degree of agreement. Formal theory 
can provide the descriptive data that can answer the 
managerial question "What is the payoff of information 
systems?" 



• Explanation — Formal theory consists of a set of 

interrelated propositions such that some can be deduced 
from others and, together, they make sense out of what 
is observed (Homans, 1964) . To say that a phenomenon 
has been explained is to argue that its occurrence in 
the future can be predicted. Explanation assumes that 
the causality problem has been resolved: that 

covariance, time order and the exclusion of rival causal 
factors have been established. 

• Prediction — Prediction is the test of explanation. If 
we claim to have explained why a given set of variables 
occurs together we must be able, by definition, to 
predict their future relationships. 



2. What a Formal Theory of System Implementation Is Not 

A formal theory of system implementation is something 
apart from what is usually described colloquially by the term 
"theory . " 

a. A model is not a theory. Models are sets of relationships 
built upon assumptions. Some of these assumptions may be 
based upon rigorous empirical assessment or upon an 
empirical base of everyday experience. But a model itself 
is a metaphor; it is used as an aid to description. 
However, a model is not a foundation for deriving 
testable hypotheses from interrelated propositions. 

b. Conceptual frameworks are not theories. 

c. A single proposition, say, if X then Y is, by itself, 
not a formal theory. 

d. Nor do sets of propositions, by themselves, constitute 
theory. They must be arranged in a deductive scheme 
(Homans, 1964 ) . 

e. Taxonomies are not theories although they may be a 
starting point in the development of propositions that 
can lead to formal theory. 
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f. Criticism, journalistic observation or hunches are not 
theories . 

D. ELEMENTS 07 A FORMAL THEORY 07 SYSTEM IMPLEMENTATION 

1. Formal Thaory as Intarralatad Propositions 

A formal theory consists of a set of testable 
propositions that are related in such a way that some are 
deduced from others. That is, for example, Proposition Three 
may be deduced from Propositions One and Two while Proposition 
Four can be deduced from Propositions Two and Three. 

2 . Tha Function of Propositions in Building Formal Thaory 

The propositions that are the first level of analysis 
in theory construction are themselves statements of 
relationships between the next level of analysis, the concept. 
Concepts are the bricks in the building of formal theory held 
together in propositional statements. 

In the proposition If X then Y X and Y are the 
concepts. The use of concepts is critical in the development 
of a formal theory. A formal theory, like the proverbial 
chain, is only as strong as its weakest concept. Concepts 
represent pieces of reality. They have three functions: 

a. Introduce new points of view into a scientific process. 

b. Convert perceptions of an environment into scientific 
activity . 

c. Make possible deductive reasoning. 

3 . Concepts as the Basis of Propositions 

Through concepts, everyday experience can be raised 
to the level of shared knowledge. Nonetheless, a specific 
concept is a commitment to a single viewpoint, excluding all 
others by its formulation. A concept is also a plan of action 
to observe a phenomenon. Concepts suggest operational 
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activity and measurement. At the conceptual level, as shown 
in Figure 1, a propositional statement takes the form if A, 
then B. If user involvement is high, then a system 
implementation will be successful. 

This statement is put into a testable form at the 
operational level. It is expressed in the form that a high 
score on an attitudinal scale indicating user involvement 
leads to a high score on an attitudinal scale indicating user 
satisfaction. A researcher proposes a null hypothesis that no 
relationship exists between the scales. Specifically, that, 
say, a Pearson correlation of the reported scores will be 
zero . 

The alternative hypothesis will be that some 
relationship exists between the scales. Again, the specific 
practice will be to reject the null hypothesis and accept the 
alternative if the probability, of a reported Pearson 
correlation (or similar gauge of the association of two 



variables) having occurred by chance 


alone. 


is less than .05. 


Conceptual level 


Concept A 




Concept B 




User 




Successful 




Involvement 




Implementation 


Operational level 
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Indicator B 




Attitudinal 




Attitudinal 




Questionnaire 




Questionnaire 




Scale of 




Scale of 




Involvement 




Satisfaction 



Figure 1. Operationalizing Concepts 
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a. Definition of Concepts 

Defining a concept is a critical link in the 
conceptualization of theory. A definition must specify what 
is included in the concept (not what is excluded) . They should 
be expressed in scientific language, not in everyday, 
colloquial language. 

That means that the expression should have a high 
degree of consensus among its users as to what it refers. 
Moreover, a definition must not be a tautology. Rather, it 
must have an external reference. 

Jb. Nominal Definitions 

This is the primitive level of defining a concept. 
Generally, it means nothing more than substituting one word 
for another or employing pseudo-mathematical notation. An 
example would be the use of a capital letter S to be a symbol 
for the concept of self. 

Nominal definitions have no meaning beyond that 
arbitrarily given to them (Bierstadt, 1959) . No claim to 
empirical verification is possible. A nominal definition 
cannot be proven true or false because, by definition, it is 
just as it is defined (Denzin, 1970) . Being true by 
definition, nominal definitions cannot provide a basis for 
causal inference. 

At root, they are useless for the construction of 
propositions for formal theory. Nonetheless, nominal 
definitions are not without utility. The use of pseudo- 
mathematical notation at least has the virtue of economizing 
on space in published matter. 

Further, a nominal substitution of one term for 
another that is burdened with emotional baggage allows a 
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concept to be treated in a manner that is more scientific and 
less ideological. Finally, nominal definitions can be a 
medium by which new terms and concepts are introduced into 
scientific usage. 

c. Real Definitions 

A real definition resolves a concept into its 
constituent parts (Denzin, 1970) . For example, user 
involvement can be defined as the users of a system that 
participated in its development and implementation. This 
definition can be looked at in terms of users, system, 
participation, development and implementation. It points 
immediately to specific steps for observation. 

The attitudinal scale used in an example earlier 
would need to have sets of items that measured the degree to 
which a respondent was (1) a user, (2) used the system being 
evaluated, (3) participated in development and (4) 
participated in implementation. 

Moreover, a set of items would also be needed to 
establish the meaning of the term "participation" to each 
respondent. The test of a real definition is made in terms 
of the empirical observations it can generate. 

d. Operational Definitions 

An operational definition prescribes what to do to 
observe a concept. It is the vehicle by which real definitions 
are translated into empirical tools. It is tantamount to an 
empirical indicator at the operational level of a proposition. 
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B. CONDITIONS FOR ESTABLISHING CAUSAL INFERENCE 



1 . Th« Function of Causal Infaranca in Building Formal 

Thaory 

The foundation of formal theory is a set of testable 
propositions that, finally, can lead from explanations to 
predictions. That implies that concepts in a proposition 
must be expressed in causal order. The proposition, if 
Concept X, then Concept Y requires that Concept X be the cause 
of Concept Y. 

2. Three Conditions for Establishing Causality 

For Concept X to be established as the cause of 
Concept Y, three conditions must be met: 

a. Covariance: both X and Y must vary. 

b. Time order: X must vary first followed by variance in Y. 

c. Rival hypotheses that can explain any observed variance 
in Y must be excluded. 

a. Time Order 

Establishing time order between two variables 
requires that the design of an empirical investigation must 
account for time. Evidence must be gathered at more than a 
single point in time. This suggests a longitudinal or time- 
series design. If the effect of an intervening or independent 
variable X upon a dependent variable Y is of interest, then 
the design must involve collection of data before the 
appearance of X (observation one: 01) and again following the 
intervention (observation two: 02) . A pretest observation and 
a posttest observation around the occurrence of the 
intervening variable X is expressed in this notation: 

01 X 02 
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A difference between 01 and 02, ceteris paribus may be 
attributed to the intervening effect of X. 

b. Rival Explanations for Obamrvad Changma 

Things, of course, are not otherwise equal. 
Alternative explanations arise to account for differences 
between 01 and 02. These rival explanations are represented 
by the threats to internal validity (Campbell and Stanley, 
1966) and by the particular techniques by which data are 
collected (Denzin, 1970; Phillips, 1971) . 

Internal validity deals with the issue as to 
whether reported findings are due to the effect of an 
experimental variable or due to other causal factors for which 
no accounting was made in the design of the research. Some 
threats to internal validity deal with spurious conclusions 
based upon observations made before and after the effect of 
an intervening experimental variable. Others deal with 
erroneous conclusions based upon supposed differences between 
comparison groups. 

The threats to internal validity are (Campbell and 
Stanley, 1966) : 

- History effects. 

- Subject maturation. 

- Re-test sophistication. 

- Changes in instrument calibration. 

- Statistical regression. 

- Bias in selection of subjects. 

- Differential subject drop-out effect. 

- Interaction of selection bias and maturation effect. 

The content and affect of each of these threats are explained 
in detail in Chapter III. 
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The techniques by which a researcher collects data 
can have a similar effect on his or her conclusions. A study 
might report an observed level of user satisfaction with the 
implementation of a new system, ascribing this finding to the 
effect of experimental variable X, say a program to involve 
users in the system's design. Yet, in reality, the observed 
difference might be caused by the very wording of the 
questions in a questionnaire used to measure satisfaction. The 
order in which the questions appear in a questionnaire or the 
graphic format of the questionnaire itself could also create 
the observed effect. 

The difficulty with the ways in which researchers 
collect data is, according to Phillips (1971) that, ” ...our 
measurements of most variables are caused less by the 'true' 
values of those variables than by the effects of measuring 
them." If the three conditions of covariance, time order and 
the exclusion of rival explanations are met, then a researcher 
can confirm that a Concept X is the cause of a Concept Y. If 
the study of system effectiveness can establish causal 
inference by generating causal propositions, then it has the 
foundation for constructing a formal theory of system 
effectiveness . 

If students of information systems can construct 
a formal theory of system effectiveness, then the discipline 
of MIS will be able to describe, explain and predict the 
conditions under which an implementation of an information 
system is likely to be a success or a failure. If an academic 
discipline is not engaged in the construction of formal 
theories that describe, explain and predict, what then is it 
doing. If a field of inquiry is not engaged in doing science, 
it risks being condemned to impotency and irrelevancy. 
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II. LITERATURE REVIEW 



A. COST-BENEFIT ANALYSIS 

Findings on whether or not Cost-Benefit Analysis should 
be used, the problems and weaknesses associated with using it 
and the methods by which it should be conducted are summarized 
below . 

Ives, et.al (1983), states that: 

Theoretically, the determination of information system 
value is a matter of economics: the costs of system 
operation and development are subtracted from the actual 
benefits (in organizational effectiveness) to obtain the 
net value of the system to the organization. 

In practice: 1) . intangible costs and especially benefits 
of the information systems are difficult to recognize and 
to convert to their monetary equivalent; 2) . Some DSS are 
used for disparate, relatively unstructured, ad hoc 
decisions; objectively assessing the benefits of such 
systems may be nearly impossible; 3) . Data on system 
success may be determinable but not recorded by the 
organization and, therefore, unavailable for research 
purposes . 

Keen (March 1981), suggest that traditional cost-benefit 
analysis is not well-suited to Decision Support System. 
However, he notes that it is effective for many computer-based 
systems. Cost-benefit analysis is sensitive to assumptions 
such as discount rates and residual value. 

If cost-benefit analysis is appropriate or can be applied, 
there are problems or weaknesses. Chervany and Dickson 
(1970) note that, a major source of difficulty is the failure 
to view Management Information System development and evalua- 
tion as a problem in decision-making. Keen (1981) identifies, 
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a general weakness of the cost-benefit approach as being its 
requirement for knowledge, accuracy and confidence about 
issues which for innovations are unknown, ill-defined and 
uncertain. The complex calculations of cost-benefit analysis 
are replaced in value analysis by rather simple questions 
about usefulness. 

King and Schrems (1978), highlight three problems with 
using cost-benefit analysis for evaluating decisions about 
information system in organizations. First, the difficulties 
of assigning cost and benefits, second, the failure to specify 
the critical characteristics demanded of the system and 
finally, social and political realities. 

There are several methods suggested in the literature for 
conducting cost-benefit analysis; the most common methods are 
discussed in the following paragraph. Chervany and Dickson 
(1970), conclude that: 

It is the prime responsibility of the MIS specialist to 
synthesize these three elements - controllable variables, 
constraints, and uncontrollable variables - and project 
payoff functions (costs and benefits) for the alternative 
MIS configurations under consideration. 

They further note that cost-benefit analysis are the 
results of decisions, of not time; therefore, techniques of 
economic evaluation of management information systems must be 
decision-activity oriented. 

B. USER SATISFACTION 

This section summarizes comments and observations from 
several authors on user satisfaction. First, a discussion on 
productivity gains that can be addressed by Management 
Information System is presented. Next, effectiveness and 
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efficiency for users information system are discussed. 
Finally, some of the expected payoffs and costs are des- 
cribed. 

Chandler (1982) states: 

IS can be evaluated from two different perspectives: 

• Focus on the computer system domain, where performance 
is measured in terms of resource utilization, cost and 
efficiency . 

• Focus on user domain throughput, where reliability and 
response time are common measures. 

An information system can be viewed as a symbiotic 
relationship between the users of the system and the system 
itself. Ideally, an information system should be evaluated 
with equal consideration given to both user constraints and 
to system constraints. 

1 . Productivity 

Amerrezvani (1986) said, "the recent explosion of 
information technology has loosed a Pandora's box of questions 
about how computers and electronic communications can be used 
effectively in the workplace, however, current uses of PCs 
don't always contribute markedly to productivity". 

Olson and Lucas (1982) point out that, "Office 
Automation is expected to increase organizational productivity 
through redefinition of office work rather than increase 
efficiency of current office functions." Additionally they 
state that, "Automated office systems can provide a powerful 
mechanism for increasing productivity and improving the 
quality of work life by changing the fundamental nature of 
organizational information processing." 

Finally, Strassmann (1985) concludes that: 



15 



The cost of automating office is a matter of people not 
a matter of machines. OA projects are dominated by user 
costs, or organization cost. Due to heavy initial orga- 
nizational costs, the benefits from most investment in IT 
can be realized in the long run. Realizing a long-term 
payoff from IT depends on the gain one obtains from people 
who learn how to work more effectively. 

The primary objective of such projects is to rapidly make 
people much more productive. And do not automate 
obsolete patterns of work, because you will not get the 
full benefit of the technology. You will only get minor 
gains and minor gains soon will no longer be sufficient 
to maintain a competitive edge. 

2. Effectiveness/Efficiancy 

Ives, et.al. (1983), state that: 

DSS can be judged on two criteria: efficiency and 
effectiveness. Efficiency deals with how well they 
do what they do. While effectiveness takes a broader 
focus. Are they doing the right thing? Are new 
computer technologies being successfully integrated 
into the organization? 

Strassmann in (1985) "Making Office Systems Pay Off" 
(1985) , prefers to measure effectiveness - doing the right 
thing. Effectiveness relates to the operation of groups, not 
individual, and effectiveness is judged by customer 
satisfaction. A higher level of payoffs come from improving 
group effectiveness and efficiency. 

3. Payoffs/Costs 

Strassmann (1985), states that: 

The goal of office automation from the point of view 
of many top managements, is to reduce employee head- 
count or to reduce direct expenses. These expenses 
include telephone charges, courier service, travel, 
etc. Management wants bottom line payoffs. Office 
automation can no longer be justified by promoting 
the use of technology with the vague claim that it 
will "increase productivity." 
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Levels of payoffs . There appears to be a progression 
of potential payoffs from installing office automation 
systems. The lowest level payoffs come from 
automating individual tasks, aimed at reducing head- 
count. These payoffs probably will not be great 
because they involve individual tasks and individual 
employees. A higher level of payoffs come from 
improving group effectiveness and efficiency. 

Getting a financial payoff from automated office 
systems will not be easy. New technology, by itself, 
is not the answer. The big payoffs will come from 
solving the "people" problem. 

Additionally, Strassmann in (1985) "The Real Cost of 
OA", identifies six cost and payoff characteristics associated 
with Office Automation (OA) . The prime responsibility of 
management on information technology projects is to cope with 
organizational issues rather than with technology issues. 

• Organized labor is becoming keenly aware of the negotia- 
ting leverage that mismanaged OA projects hand over to 
them. 



• The cost of any activity that is labor intensive tends 
to rise faster than the general rate of inflation. 

• Sending employees to classes on company time at a 
company training center is the most expensive way of 
learning . 

• Most importantly, the risks of run-away organizational 
costs due to technical system failures cannot be 
tolerated . 

• To create the right environment for sharing organization 
learning, it will be necessary to create new standards 
for access to information sources. 

• Until we have better insights into the behavioral 
aspects of people, organizational costs will be high. 
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Olson and Lucas (1982), state that: 

The political view assumes that automation implies a 
division of labor that leads to decrease in skill, 
knowledge, and worker control over activities. 

C. OTHER MODELS 

This section discusses other models or perceptions 
collected in the nonempirical literature review. They are 
broken-down into pieces that address productivity, effecti- 
veness/efficiency and other general perspective. 

1 . Productivity 

Borko (1983) defines productivity as the ratio of the 
number of units of output to the number of units of input. 
Productivity can be measured by converting input and output 
units to their respective dollar equivalents. He further 
states that: 

Productivity rises when the number of units of output 
increases while input holds steady; or conversely, when 
the number of units of output remains fixed and the number 
of units of input decreased. 

Obstacles to measuring knowledge worker productivity: 

• Defining the nature, the value, and the unit of measure 
for the output of the knowledge worker. 

• Tendency to deal with only the quantifiable and not with 
the quality of the outputs. 

• The work done by a knowledge worker in a current time 
period may not show results until several periods later. 



Carlyle (1987), states, "The difficulty of measuring 
productivity gains has also deterred potential buyers, but 
some vendors, like Texas Instruments, are in it for the long 
haul." He further notes that, at a time when the industry 
is paying lip service to the need for Computer Aided Software 
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Engineering (CASE) , computerized tools for software developers 
are emerging ahead of an accepted productivity index or 
effective measure. Many large companies are afraid that they 
could invest more in trying to improve productivity than they 
would actually be able to save. 

Mason, et.al. (1984) said, "In the area of public 
productivity and computer use, computers serve three purposes; 
reducing costs ... increasing revenues ... most impor- 

tant .. .providing better services." 

2 . Ef f«ctiv®n«**/Ef f ici*ncy 

Hamilton and Chervany (1981), contend that, effec- 
tiveness is determined by comparing performance to objectives. 
The way to assess system effectiveness is to: 

• Determine the task objective of the system or the 
organizational units [using] the system and to develop 
criterion measures to assess how well the objectives 
are being achieved. 

System effectiveness is determined by attainment of a 
normative state, e.g., a standard for "good" practices. 
Effectiveness is conceptualized in terms of resource viability 
rather than in terms of specific task objectives. Evaluation 
of system effectiveness is difficult due to its multi- 
dimensionality, its quantitative and qualitative aspects and 
the multiple, and often conflicting, evaluator viewpoints. 

The assessments of management information system 
effectiveness are often controversial and can be sources of 
disagreement and conflict between different functional groups 
involved in MIS implementation: users, management information 
system developers, internal auditors, and top management 
personnel. Maggiolini, (1986), identifies four points that 
can improve office automation technology efficiency by: 
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• Automating all or some parts of the office information 
process; 

• Eliminating some of the transformations of medium; 

• Eliminating or reducing the shadow activities/functions; 

• Speeding up the information process itself. 

3 . General Perspective 

Hamilton and Ives (1982) , said that, "the key to good 
research, though is not just in choosing the right research 
strategy, but in asking the right questions and picking the 
most powerful method (s) for answering questions given the 
objectives, research setting and other salient factors." 
Ginzberg (1978), states that: The key point measure of 

effectiveness depends upon the intent of a project its goal. 
We can develop a behavioral measure of Operation 
Research/Effectiveness if we explicitly address the issue of 
goals at the start of the project. Finally, Strassmann 
(1976), suggests nine steps to better management of Management 
Information System: 

a. Identify costs 

b. Keep scores on unit costs 

c. Establish standard costing 

d. Set up accountability centers 

e. Apply competitive pricing 

f. Plan for the long term 

g. Let the users control 

h. Deemphasize the technology 

i. Use job enlargement 
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III. METHODOLOGY 



A. SELECTING STUDIES 07 SYSTEM EFFECTIVENESS 

1. Choosing a Medium from Which to Select Studies 

Published articles were chosen as the source from 
which to select the studies of system effectiveness to be 
reviewed in this thesis. Published transactions of 
conferences were excluded for three reasons: 

a. The difficulty of obtaining comprehensive bibliographical 
search coverage of all conferences that hold sessions 
related to system effectiveness. For example, an on-line 
bibliographic search of computer and management 
information system (MIS) failed to cover the important 
annual International Conference on Information Systems. 

b. No evidence exists to support a decision that papers 
presented at a conference that publishes its transactions 
are superior to a conference that does not. Leaving 
aside the problems of obtaining reliable bibliographic 
coverage of conferences that do publish transactions, to 
include them would unfairly exclude papers from non- 
publishing conferences. 

c. Both of the preceding issues are moot in that presenting 
a paper at a conference is, by convention, a step to 
submitting it for publication by a journal. Indeed, 
papers accepted by the editors of the International 
Conference on Information Systems convey the best of them 
to journal editors for consideration as published pieces 
(Gallegos, 1985) . In short, conference papers worthy of 
publication will appear in journal omitting conference 
papers prevents redundant entries. 



21 



2 . Choosing Journals from Which to Salsct Studios 

This review of academic studies began with a selection 
of journals from which reports of these studies would be 
taken. All reports of studies of system effectiveness in 
English-language journals in the field of management 
information systems (MIS) and the related field of management 
science were surveyed. We took as an authority for choosing 
this set of journals the works of Davis (1980) , Hamilton and 
Ives (1983) and Vogel and Wetherbe (1984) . A consensus of 
journal names from these studies yields the following list: 

a . Academy of Management Journal 

b. Accounting Review 

c . ACM Computing Surveys * 

d . Administrative Science Quarterly 

e . Communications of the ACM 

f . Data Base 

g . Datamation 

h . Decision Sciences 

i . EDP Analyzer ** 

j . Harvard Business Review 

k. IEEE Computer *** 

l . Information and Management 

m. Interfaces 

n . Journal of Management Information Systems 

o . Journal of Systems Management 

p . Management Information Systems Quarterly 

q . Management Science 

r . Omega 

s . Sloan Management Review 

* ACM: Association for Computing Machinery 
** EDP: Electronic Data Processing 

*** IEEE: Institute of Electronic and Electrical Engineering 
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Hereafter, this list is known as the consensus list. To it, 
was added Public Productivity Review which was discovered 
during a search for periodicals related to the issue of 
productivity. While the preferred publication outlets on the 
consensus list are refereed journals written by and for 
academic members of the MIS community, the list includes 
periodicals that are aimed at an audience of MIS practitioners 
and some that are not journals per se but are commercially 
published trade publications. 

3. Choosing a Tima Span from Which to Salact Studies 

The work on this review began in January, 1989 and was 
completed by June, 1989. The last journal articles that were 
considered for inclusion were published with the calendar 
period of 1988. 1970 was chosen as the beginning point for 
on-line and manual bibliographic searches. 

However, no time limit was placed on inclusion of 
studies that were discovered by virtue of references in 
articles found in the 1970 to 1988 range by manual or on-line 
searches. The assumption was that articles worthy of 
inclusion that were published prior to 1970 would be 
referenced in articles published after 1970. 

4. Searching for Studies of System Effectiveness 

a. On-Line Bibliographic Search 

In January, 1989, the Dialogueue access station in 
the Dudley Know Library of the Naval Postgraduate School was 
used for a bibliographic search based on key words in the 
titles and abstracts of articles. The following search key 
was constructed: 
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(Productivity or Effectiveness) and (Measures or 
Measurement or Assess or Assessment) and Information 
Systems . 

This search identified 154 items that matched the 
search key. Abstracts on the 128 of these that were published 
in English were requested. Out of that set, 51 items were 
identified that dealt with the productivity or effectiveness 
of information systems. The other 77 items dealt with the 
monitoring of computer performance, productivity of software 
development or topics unrelated to system effectiveness but 
nevertheless included some combination of the search key words 
in their titles or abstracts. 

Of the 51 items related to system effectiveness, 
26 were conference papers. Out of the 25 items dealing with 
productivity or effectiveness of information systems that were 
journal papers, ten were in journals on the consensus list. 
This search covered several important conferences related to 
information systems, including some of which we would not 
otherwise have known. 

Nonetheless, as mentioned earlier, the flagship MIS 
conference, the annual International Conference on Information 
Systems, was not represented. In retrospect, this on-line 
search was naive in its conception and incompetent in its 
execution. During the manual bibliographic search, it was 
discovered that the "productivity" and "effectiveness" 
keywords were scarcely comprehensive in the coverage of the 
literature of system effectiveness. 

The keywords that authors in this literature 
applied to their works included: 

- Cost 

- Benefit 
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- Cost Benefit 

- Evaluation 

- Success 

- Utilization 

- Performance 

Moreover, the search key applied only to Dialogue file 13, 
which is the INSPEC data base for the period from 1977 to 
1988. That this file began its coverage at 1977 rather than 
1970 already meant that the original intention was 

compromised. A different INSPEC file contained items 
published before 1977. That file would have been included in 
the search but for a funding limit on access to bibliographic 
search services. 

Subsequent experimentation with on-line 
bibliographic searching revealed that other databases in the 
Dialogue set might have yielded more articles. Certainly, by 
not searching beyond the INSPEC file, which covers the 
computer and MIS fields, publications in fields such as 
general management (Harvard Business Review, Sloan Management 
Review) and management science (Management Science, Decision 
Science, Administrative Science Quarterly) were missed. 

Because limited funding prevented further on-line 
bibliographic searching with a wider net of key words applied 
a broader set of databases, this study turned to manual 
searching . 

b. Manual Bibliographic Saarch 

We manually examined the Periodicals on the 
consensus list for the years 1970 to 1988 were manually 
examined (although some of these periodicals were founded 
after 1970) . Indices of article titles for the covered years 
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were examined or every issue of a periodical (where indices 
were not available) were examined. The manual search sought 
titles of articles that seemed to deal with the effectiveness 
of information systems. This included articles on the 
performance of systems in specific industries or by type of 
specific application (such as decision support systems) as 
well as information systems in general. This effort yielded 
93 articles purporting to deal with the measured effectiveness 
of information systems. 

The manual search, not surprisingly, found the same 
ten titles that were revealed by the on-line search. The 
number of articles identified in the on-line search compared 
to the far greater number turned up by the manual effort was 
dismaying. This was the point at which it was discovered how 
hopelessly limited had been the key word synonyms for 
productivity and effectiveness that were used. 

c. References from Journal Articles 

Reference lists at the end of articles found 
manually were used to track down important works that either 
were published before 1970 or whose relevance was disguised 
by its titles. This procedure produced an additional 31 
articles. These included works that dealt with system 
effectiveness as well as articles that dealt with 
productivity, cost-benefit analysis, organizational 
effectiveness and research methods that were written for a 
broader audience than the MIS discipline. 
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d. Gathering of Article Reference by Other Mem. na 

When studying a particular topic, a researcher 
becomes sensitized to its appearance in any form. Discussions 
of system effectiveness or productivity in popular or trade 
publications, that might otherwise go unnoticed, will now leap 
into a researcher's awareness. Even colleagues who are 
peripheral to a study become sensitized as well. They offer 
references and clippings dealing with productivity and the 
effectiveness of information systems. By these processes the 
inventory of works on productivity, system effectiveness and 
the measurement of effectiveness in this study was built. 

A good deal of these were from popular and trade 
publications. They included magazine articles, newspaper 
items, student theses, books, transcripts of speeches and 
promotional literature on software and consulting services. 

5. Sorbing the Works on System Effectiveness 

In combination, the preceding methods yielded an 
overall list of 118 references that was initially considered, 
by virtue of their titles, for review in this thesis. They are 
listed in Appendix A. 

a. Empirics! Studies 

A reading of the 118 articles in Appendix A 
identified 45 empirical studies of system effectiveness. 
These 45 articles report empirical studies of the 
effectiveness of an information systems or an aspect of 
research techniques for studying system effectiveness. 

These articles are distinguished from others in the 
original set of 118 in that they report findings based on data 
collected specifically for the study being reported. These 
are the studies that this thesis evaluates. They are studies 
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are listed in Appendix B. Among these 45 were articles that 
did not deal directly with a study of a system implementation 
but rather were concerned with the development of a research 
tool to study implementation success. These were retained 
because their study research techniques contributes to how we 
know what we know about system effectiveness. 

Excluded from this review were studies that dealt 

with : 

1. Computer performance. 

2. Software development. 

3. Other topics not related to information system 
effectiveness . 

b. Non -Empirical Works About the Study of System 

Effectiveness 

Excluded from this review of empirical studies were 
articles which did not report findings of a concrete study of 
a system implementation. These included essays on system 
effectiveness, literature reviews of system effectiveness or 
prescriptions for the methods of studying information system 
effectiveness. Appendix C lists the 22 articles in this 
group. They are, of course, the references that were 
discussed in the preceding chapter which reviewed the MIS 
literature on the assessment of system effectiveness. 

c. Miscellaneous Literature 

The remaining references from the initial 118 were 
categorized as: 

Popular and trade literature. 

- Literature dealing with productivity or organizational 
effectiveness in general but not specifically related to 
MIS. 
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Works dealing with research methodology without specific 
application to MIS. 

These are listed in Appendix D. 

B. REVIEWING THE EMPIRICAL STUDIES 

The 45 empirical studies (listed in Appendix B) were 
individually evaluated for: 

1. Design of data collection 

2. Handling of threats to internal validity 

3. Handling of threats to external validity 

4 . Time dimension 

5. Data collection methods 

6. Definition of effectiveness 

1 . Design of Data Collection 

From a reading of the descriptions of research 
methodology and data collection techniques in each study, it 
can be determined if it: 

a. Employed an explicit, experimental variable. 

b. Gathered data before the administration of an 
experimental variable. 

c. Gathered data after the administration of an experimental 
variable . 

d. Gathered data from comparison groups at the same points 
in time . 

e. Whether the membership in comparison groups was 
randomized . 

Knowledge of these five elements permitted classification of 
each study according to its research design using the Campbell 
and Stanley (1966) taxonomy. 
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2 . Handling Threats to Internal Validity 

Once a study was classified according to the Campbell 
and Stanley taxonomy, it was possible to assess its ability 
to handle each of the threats to internal validity listed 
below: 

a . History 

A history effect involves events occurring between 
a pretest observation and a posttest observation which affect 
an observed phenomenon along with an intervening variable X. 
Designs that collect data at both pretest and posttest points 
but lack a parallel control group cannot control for the 
effect of history on a study's findings. Designs lacking a 
pretest point but involving a comparison group, while weak in 
other respects, can control for the effect of history. 

b. Maturation 

Maturation refers to changes within the respondents 
themselves that can account for differences in measurements 
between the first and second observations. As with the history 
effect, designs that collect data at both pretest and posttest 
points but lack a control group cannot control for the effect 
of maturation. Where a design lacks a pretest point but 
involves a comparison group, the capacity of the design to 
exclude maturation as a rival explanation of its findings in 
not clear from a design alone (as in the case of the static 
group design) . Designs that use randomized comparison groups 
(such as the posttest only control group) are not affected. 
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c. Re- test Sophistication 

Re-test sophistication arises when respondents 
become skilled at taking a test, filling out a questionnaire, 
answering in an interview or taking part in an experiment. As 
much as the intervening variable, such changes in skill can 
cause changes in readings at a second observation. Where a 
design lacks a pretest phase, yet uses comparison groups, this 
effect is controlled. Designs that involve a pretest and 
posttest but lack a comparison group (such as the one group 
pretest/posttest design) are threatened by re-test 
sophistication, For other designs, the effect is neither 
controlled nor threatening. 

cf. Instrument Calibration 

Where people are the instruments of recording and 
measuring, changes within them can produce changes in the in 
reports of events. Examples of human re-calibration are 
fatigue, growing skill at conducting interviews, boredom or 
developing self confidence as observers. This effect is 
controlled in designs that involve comparison groups with 
randomized assignment of members or, lacking that, have no 
pretest stage. The one-group pretest/posttest design is 
vulnerable to instrument calibration effects. 

a. Statistical Regression 

Where comparison groups in a study of system 

\ 

implementation have been selected precisely for their extreme 
differences in scores on tests or surveys before a study, the 
statistical regression toward the means score is likely to 
occur on the posttest observation. Generally, respondents with 
extreme scores on one occasion will not produce the same 
extremes on another occasion. A design alone does not tell 
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whether statistical regression affects a particular instance 
of a one-group pretest/posttest design. The static group 
design, however, is vulnerable to is effect. Designs involving 
randomized comparison groups are able to control for the 
effect of regression. 

f. Selection Bime 

Selection bias is a rival explanation for a study's 
findings when comparison groups assumed to be equivalent are 
actually different due to biases operating during the 
selection of respondents. These differences can account for 
the differences in reported data between the two groups. Any 
design with a pretest and a posttest can control for this 
effect. Those lacking a pretest or a randomized comparison 
group are vulnerable to selection bias . 

g. Respondent Mortality 

Between non-randomized 'comparison groups, there 
can be a selective drop-out of participants from a study while 
researchers assume that dropping-out is random. This 
differential drop-out effect can account for differences in 
data reported from the groups. The designs that control for 
this effect and those that are vulnerable to it are identical 
to that of the effect of selection bias. 

h. Selection-Maturation and Other Interactions 

In multi-group research designs, the interaction 
of a selection bias with a maturation effect could be mistaken 
for the effect of an intervening experimental variable. 
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3 . Handling Threats to External Validity 

As with the threats to internal validity, knowledge 
of a study's placement in the Campbell and Stanley (1966) 
taxonomy of research designs allowed researchers to assess 
its capacity to exclude threats to external validity. 

a. Interaction of Tmating and an X Variable 

In certain kinds of studies, the experience of 
participating in an experiment, completing a questionnaire, 
submitting to an interview or being observed at pretest stage 
sensitizes a group of respondents to an X variable. That is, 
by being subjects in a study, these people have a heightened 
sensitivity to an X stimulus that is not characteristic of the 
general population. 

To that extent, the findings of such a study are 
not generalizable beyond the sample group. The one group 
pretest/posttest design and the true experimental design are 
both vulnerable to this effect. Groups that have some elements 
of non-pretest comparison such as the posttest only control 
group and the Solomon four-group design are able to control 
for it. Other designs are not affected either way. 

b. Interaction of Selection and the X Variable 

Campbell and Stanley (1966) use as an example of 
this threat a researcher seeking a school district to provide 
a sample of students for a study. Nine districts turn down the 
researcher. A tenth district agrees to participate. Surely, 
in some way, the tenth differs from the other nine. The 
results that this study yields will, therefore, not be 
generalizable to a large population of students per se. 

To the extent that an experimental variable in this 
study has differential effects between an experimental group 
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and a control group, it is likely that characteristics of the 
school district caused the X variable to be more successful 
than it would begin other school districts. 

c. Reective Arrengemente 

The artificiality of an experimental setting or of 
questionnaire choices put respondents in a Guinea-pig, I-am- 
special role. Whatever behavior that is thereby observed or 
recorded does not represent how these same people are likely 
to act in everyday life. 

d. Multiple X Interference 

While a conventional research design involves a 
single administration of an X variable to experimental group, 
some quasi-experimental designs such as the equivalent time 
samples design, and the equivalent materials samples design, 
involve repeated administrations of an X variable to one group 
of respondents. 

The trouble is that the effects of prior 
administrations remain with the respondents, building up with 
each subsequent administration. Such groups undergo an 
experience as research subjects that has few counterparts in 
everyday life. Thereby, they are rendered unrepresentative of 
the larger population. 

4 . Time Dimension 

The Campbell and Stanley (1966) taxonomy of research 
designs also provides a means to assess whether a study has 
taken account of the time dimension as a basis for 
establishing causal inference. 

Studies employing pretest and posttest observations 
and those that employ time series multiple observations, by 
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definition, account for the time dimension. Studies involving 
an observation at a single point in time do not. 

5. Techniques of Data Collaction 

By convention, an empirical study is obliged to 
describe the technique by which it gathers data. That 
convention was upheld by all 45 of the studies reviewed here. 
Each study was classified according to the following taxonomy 
of data techniques: 

a. Self-completion questionnaire, administered under the 
supervision of the researchers. 

b. Self completion questionnaire, mailed out to subjects to 
be returned by mail. 

c. Interview questionnaire. 

d. Telephone interview. 

e. Laboratory experiment. 

f. Non-obtrusive technique: archival records. 

g. Review of observations by expert panel. 

€. Definition of System Effectiveness 

Each study was read for its definition of how a system 
would be judged to be effective. No taxonomy of such 
definitions exists a priori The classification categories 
emerged in the process of reviewing the studies. 

7 . Degree of Change in Effectiveness Reported in the 

System 

This thesis suggests that the appropriate enterprise 
for academic research in MIS effectiveness is the construction 
of a formal theory of system effectiveness. A theory of system 
effectiveness necessarily involves a knowledge of the 
determinants of such effectiveness. To understand the 
determinants of effectiveness, a researcher must know by how 
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much the effectiveness of an organization was improved or 
degraded by the implementation of an information system. 

Therefore, it is appropriate to inquire as to the 
level of changes in effectiveness that are reported in each 
empirical study of system effectiveness. If doubts about the 
success of information systems are harbored by senior line 
managers, then a knowledge of the reality of information 
payoffs is a contributes to the current discussion. What do 
these 45 academic studies say about the successful application 
of information systems? 

Each study was reviewed for indications of the level 
of reported changes in system effectiveness. 
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IV. FINDINGS 



Appendix E contains an annotated description of each of 
the 45 studies evaluated here. Appendix F contains the 
detailed tables of findings for each aspect of the studies. 
The studies are listed in chronological order by date of 
publication . 

A. DESIGN OF DATA COLLECTION 

In Table F-l (Appendix F) , the number at designations at 
the top of the columns refer to evaluation elements in 
assigning each study to a design category according to the 
Campbell and Stanley (1966) taxonomy as follows: 

1. The study collected data in a pretest stage. 

2. The study employed a comparison group. 

3. Respondents in the study were assigned to comparison 
groups on a randomized basis. 

1 . Non -Experimental Designs 
a. Onm-Shot Case Study 

Table 1 distills the distribution of research 
designs in these 45 studies out of the detail of Table F-l 
(Appendix F) . Table 1 shows that the one-shot case study is 
the, research design of choice in MIS studies of system 
effectiveness, by a two to one margin. A one-shot case study 
has no explicit administration of an intervening, experimental 
variable (X) . Rather, as the brackets around the X in the 
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design notation indicate, the subjects from whom data was 
collected, at the sole observation point 0 3 , were assumed to 



TABLE 1 

DISTRIBUTION 07 RESEARCH DESIGNS* 



This Design 
as a 

Campbell Percentage 

& Stanley of Studies 

Research Design: Design Notation Reviewed : 



One Shot Case Study 



[X] 0 X 



66.7% 



Static Group Comparison 

X O 

0 

One Group Pretest /Posttest 

0 1 X o. 

Experiment 

R O X 0 

R 0 0 

Posttest Only Control Group 

R X O 

R 0 

Time Series 

0 0X00 



17 . 8% 



4 .4% 



4.4% 



4.4% 



2 . 2 % 



‘These percentages sum to 99.9% due to rounding errors. 



have been exposed to the effects of a non-existent 
experimental variable. In place of explicit exposure to an 
X experimental variable, the subjects at 0 3 are asked to 
recall something. Human recall is notoriously unreliable 
(Haberman, 1963). For that reason, the one-shot case study 
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is "the weakest form ofsurvey design" (Denzin, 1970). 
Fundamental to the gathering of scientific evidence is the 
need for recording differences (Campbell and Stanley, 1966) . 
Scientific testing requires the making of comparisons. No 
comparisons are made in one-shot studies. While one-shot 
studies often involve survey instruments that describe their 
face validity and Cronbach reliability indices in detail, all 
of that is a case of misplaced precision. 

It is another case of shuffling the psychometric 
deck chairs on the methodological Andrea Doria. In the 
estimate of Campbell and Stanley, rival explanations for the 
findings reported by one-shot studies are so numerous as to 
render them nearly useless in establishing causal inference. 

b. Static Group Comparison 

As is shown in Table 1, the next most frequent 
design in MIS research on systems effectiveness is the static 
group comparison. This design offers the virtue of some basis 
for examining differences. The difficulty, however, is in 
attributing the differences to exposure to an experimental 
variable by one of the groups. That is because membership 
in the comparison groups in this design is not a randomized 
assignment . 

The findings are vulnerable to the rival 
explanation that differences between the groups predated the 
intervention of an X variable. Where the comparison groups 
are deliberately selected for their extreme differences, as 
they were in some of the reviewed studies, the rival 
explanation gains potency. Another problem with the static 
group comparison is the possibility of biased drop-out by 
respondents in the period after the X variable intervened. 
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c. One Group Pretest/Posttest 

The one-group design has the virtue of a deliberate 
attempt to measure the differences between a group observed 
at 0 X and at 0 2 . Unfortunately, its lack of a comparison group 
leaves it subject to a distracting set of rival explanations 
for its findings. Chief among them are other effects 
transpiring between Oj and 0 2 besides the X variable. 

While it is superior to the one-shot case study, 
the only recognition the one-group design merited from 
Campbell and Stanley was that it was worth doing only where 
nothing better could be done. They employ the one-group as 
a bad example of research design that deludes researchers and 
their constituents alike by its closeness to a true 
experimental design. 

2 . Experimental Designs 

a . True Experiment 

The true experiment, having pretest and posttest 
data collection points paralleled by randomized comparison 
groups, controls for all threats to internal validity. The 
difficult with the MIS empirical studies of effectiveness is 
that only four percent of them used this design. 

b. Posttest Only Control Group 

While it lacks a pretest data collection stage, 
the posttest only design nonetheless qualifies as an 
experimental design by virtue of its randomized comparison 
groups. As with the true experiment, its difficulty is the 
rarity with which it was practiced in MIS research. 
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3. Quasi -Experimental Design 

A quasi-experimental design, a description coined by 
Campbell and Stanley, is an attempt to approximate 
experimental design in a field setting by scheduling the whom 
and when of measurement even though the experimental variable 
itself is not amenable to scheduling. 

a. Timm Smrima 

While it might appear that a time series design 
shared the vulnerabilities of the one-group pretest/posttest, 
this is not the case. The repeated measurements before and 
after an experimental variable controls for threats to 
internal validity such as maturation, re-test sophistication, 
instrument calibration, statistical regression and the 
interaction of selection bias and maturation. This renders 
it a much sounder design that the one-group. Regrettably, 
only 2% of the reviewed studies employed a time-series design. 

B. HANDLING THREATS TO INTERNAL VALIDITY 

The detailed evaluations of the handling of threats to 
internal validity by each of the 45 studies reviewed are shown 
in Table F-2 in Appendix F. A +1 in a column means that a 
study controlled the threat to internal validity represented 
by the column heading. A -1 indicates that a study was 
vulnerable to that particular threat. A question mark means 
that" the effect of the threat upon this study cannot be 
determined solely by knowledge of the research design. No 
symbol indicates that a threat has no effect on the study in 
whose row it occurs. 

Recall that internal validity asks whether reported 
findings are due to the effect of an experimental variable or 
result from causal factors not accounted for by the research. 
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Where a study can control threats to internal validity, its 
findings resist challenges to their scientific standing. 

TABLE 2 

THREATS TO INTERNAL VALIDITY 





Effect 
to Vulner- 
able 


Effect 

Cont- 

rolled 


Effect 

Not 

clear 


No 

effect 


History 


73% 


27% 






Maturation 


73% 


9% 


18% 




Testing 


4% 


29% 




69% 


Instrumentation 


4% 


27% 


2% 


67% 


Regression 




29% 


4% 


67% 


Selection 


84% 


16% 






Mortality 


84% 


16% 






Interaction of 
Selection and 
Maturation 


22% 


11% 




67% 


To the extent 


that a study 


cannot 


exclude 


the rival 



explanations for its findings posed by threats to internal 
validity, the scientific meaning of the reported data will be 
lost. The evaluations in Table F-2 are summarized in Table 
2. It reveals that, on average, slightly over 73% of the MIS 
studies of system effectiveness are vulnerable to the 
likelihood that their findings can alternatively be accounted 
for by the effects of history, maturation, selection bias and 
mortality . 

This is a consequence of the fact that two-thirds of the 
MIS studies reviewed here employ a one-shot case study design. 
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No studies were clearly vulnerable to statistical regression 
as a rival explanation. Moreover, rather few (4%) had their 
internal validity threatened by re-test sophistication or 
instrument calibration. 

The picture that emerges here is that of an academic 
discipline that cannot demonstrate with certainty that its 
empirical findings mean what they say. These findings erode 
a basis for establishing causal inference in the construction 
of a formal theory of the effective implementation of 
information systems. 

C. HANDLING THREATS TO EXTERNAL VALIDITY 

Table F-3 in Appendix F contains a detailed evaluation of 
the handling of threats to external validity by each of the 
45 studies reviewed in this thesis. The column headings at 
the top of Table F-3 (to the right of the year column) 
includes the two external validity threats arising from 
interactions between the experimental variable and 
characteristics of a study's design. 

The first of these is the interaction of re-testing and 
an X variable. The second is the interaction of selection 
bias and an X variable. The meaning of both of these are 
explained in Chapter III. The third column is the threat from 
reactive arrangements. The fourth is the threat from effect 
of multiple exposures to an X variable. The meaning of the 
+1, -1, ? symbols the same as in Table F-2 . 

Where internal validity asks whether the findings of a 
study can be trusted to mean what they claim, external 
validity deals with the extent to which the findings of a 
study are generalizable beyond the study to a larger 
population . 
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TABLE 3 

HANDLING OF THREATS TO EXTERNAL VALIDITY 

Effect to Effect Effect 

Vulner- Cont- Not No 

Threat able rolled clear effect 



Interaction of Re-Testing 
with X Variable 11% 


4% 




85% 


Interaction of Selection 
Bias with X Variable 


89% 




11% 


Reactive Arrangements 




16% 


84% 


Interference of Multiple 
Administrations of 
X Variable 






100% 



Table 3 summarizes the detailed evaluations of threats 
to external validity for each of the 45 empirical studies. 
The use of research designs in MIS empirical assessments 
determines these findings. With 11% of the reviewed studies 
employing a pretest stage, exactly 11% are found to have their 
generalizability threatened by a re-test effect. None of the 
reviewed studies involved repeated administrations of an X 
variable; therefore, none were vulnerable to multiple X 
interference . 

However, the 89% of the studies that used non-experimental 
designs (chiefly one-shot case studies) precisely account for 
the vulnerability to an interaction of selection bias and an 
X variable. External validity deals less with a fundamental 
question of scientific credibility than it does with a simple 
question of relevance. Every study reviewed here is 
vulnerable to a challenge to the generalizability of its 
findings to a larger population. Only the two posttest-only 
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studies provided a positive control over any of the external 
validity threats. 

If the findings of MIS research are not representative of 
any population beyond the studies themselves, the discipline 
is flirting with the likelihood that its core work is 
irrelevant beyond the pursuit of publication and tenure. 

D. TIME DIMENSION 

Table F-4 in Appendix F categorizes each of the 45 studies 
simply by whether its research design permitted an accounting 
of the time dimension in the collection of its data. 11% of 
the studies accounted for time, through designs such as the 
true experiment, one-group pretest/posttest and time series. 
The other 89% of the studies employ designs that collect data 
only at a single point. Most of those are of the one-shot 
case design. The capacity of a research effort to establish 
causal inference is impaired without a way to show that 
variable X occurs before variable Y. 

E. DATA COLLECTION METHODS 

Table F-5 (Appendix F) lists the type of data collection 
technique employed by each study. Two columns of methods are 
shown in Table F-5. One third of the studies in this review 
used two methods. 

In some cases, two methods bolstered each other in an 
effort at triangulation. This happened where a survey 
questionnaire was used in conjunction with system performance 
data collected by the information system itself. In other 
instances, a first method was used as pilot study, usually for 
instrument development. 
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TABLE 4 

DATA COLLECTION TECHNIQUES FOR PRIMARY METHOD 



Asking Questions 

Questionnaire 60.0% 

(Including mail out- 
mail back surveys: 22%) 

Interview 11.1% 

Phone survey 2.2% 

Not Asking Questions 

Non-obtrusive methods 17.8% 

(Archival ) 

Laboratory experiment 4.4% 

Diary 2.2% 

Panel 2.2% 



73.3% 



26.6% 



TABLE 5 

DATA COLLECTION TECHNIQUES FOR SECONDARY METHOD 

Asking Questions 80.1% 

Questionnaire 66.7% 

(Including mail out- 
mail back surveys: 27%) 

Interview 6.7% 

Semantic Differential 6.7% 

Not Asking Questions 20.0% 

Non-obtrusive methods 13.3% 

(Archival) 

Simulation 6.7% 
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TABLE 6 

DISTRIBUTION OF METHODS BY RESEARCH DESIGN 





Type of 


Research 


Desian 


True 


Post 






One- 

Shot 


Static 

Group 


One 

Group 


Experi- 

ment 


Test 

Onlv 


Time 

Series 


Method 














Questionnaire 
incl Mailed Out) 


61.3% 


71.4% 


50.0% 






100% 


Mailed Out 


29.0% 




50.0% 








Interview 


12 . 9% 


14.3% 










Phone survey 


3.2% 












Archival 


16.1% 


14.3% 


50.0% 






100% 


Laboratory 












100% 


Diary 


3.2% 












Panel 


3.2% 














100%* 


100% 


100% 


100% 


100% 


100% 




Sums 


to 99.9% 


due to 


rounding 


error 





As Table 4 shows, over 73% of the studies reviewed here use 
primary methods that involve asking people questions whether 
by self-administered questionnaire (completed in group 
sessions or in a mailed-out /mailed back mode) , personal 
interview or by phone interview. 

The problem posed by this proportion is that the focus of 
inquiry is the effectiveness of systems , not of individuals . 
Even if the objection is made that systems, after all, are 
essentially comprised of people, then the fallacy of 
reductionism is in play. It is a fallacy to study a level of 
organization by reducing it to its constituents elements. If 
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reductionism were a valid approach to studying the performance 
of organizations or systems, why stop at individuals? 
Individuals, after all, are comprised of living organs. By 
this logic, we would study information systems by studying 
intestines . 

One can follow this absurd path to molecular biology and 
beyond. While individuals together comprise systems (or 
groups or teams or organizations), there is a qualitative 
distinction between the analysis involved in studying 
individuals and in studying systems comprised of individuals. 
Systems (as with teams, groups, families or organizations) are 
not merely the sum of the discrete individuals. 

Directing questions at individuals gathers data about 
individuals but collects little about the characteristics of 
the system. This leaves aside the even more troublesome issue 
about asking people questions even when it is individuals that 
are being studied. This preference for asking questions was 
even a bit stronger among the secondary methods. 

The core concern with the people-asking methods is that 
the methods tend to create the very data they are recording. 
Far from providing a neutral, innocuous means of gathering 
observations about events, these methods provoke responses 
that, in themselves, involve role-playing, Guinea-pig effects, 
looking good, passing a test, socially desirable responses, 
fabrication, data inflation and otherwise carrying on in ways 
that are not found in everyday interactions other than with 
organizational or social superiors. These same biasing and 
distorting factors are found in the reactive setting of 
laboratory experiments. But there they are well known. 

The trouble with the survey questionnaire is that it is 
assumed to have no biasing or distorting effects when it, in 
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reality is strongly biasing and distorting (Katz, 1942; Hyman, 
1949; Wyatt and Campbell, 1950; Parry and Crossley, 1959; 
Robins, 1963; Williams, 1964; Weiss, 1968; Webb et . al , 1966; 
Sechrest, 1968; Summers and Hammonds, 1969; Hambright, 1969; 
Denzin, 1970; Phillips, 1971) . 

A final problem posed by the method of asking questions 
is that the data collected can only record the expression of 
attitudes. The asking-people methods are unable to record 
expressions of behavior. Behavior can only be observed or 
recorded by observers as archival data. While MIS researchers 
occasionally speak of dealing with "behavioral" issues or 
methods, in fact they are not dealing with behavior at all but 
attitudes . 

To use a questionnaire to ask a respondent about their 
behavior is not to gather data on behavior. Rather the 
process gathers data on a respondent's attitude toward the 
behavior above which he or she is being asked. This is a 
distinction that seems largely lost on the MIS researchers in 
the study of system success. 

Examining the association of method with research design 
holds no surprises. The non-experimental designs relied 
heavily on the methods that involve asking questions. But 
then so did all of the true experiments. However, the other 
experimental design, the posttest only control group, employed 

only a non-asking method. So did the single quasi- 

\ 

experimental design. 

A strong relationship might be expected between a weak 
design such as a one-shot case study and a troublesome method 
such as asking questions of people. Both evince a reluctance 
to commit to research efforts that involve time series or 
attention to the details of non-obtrusive methods. 
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The discussions in many of the reviewed studies echoed the 
conventional wisdom that gathering data on the costs of an 
information system is difficult enough. Attempting to gather 
data on a system's benefits is regarded as futile. Thusly, 
these researchers justify turning to methods and designs noted 
more for their convenience than their scientific rigor. 

If one has learned only survey methods as a graduate 
student, one is unlikely to explore alternative methods as 
junior faculty member under pressure to achieve tenure. This 
is especially the case where those alternative methods and 
designs take longer and are more difficult. Career expediency 
leaves causal inference in the dust. That the non-obtrusive 
archival data method is practiced so little is surprising in 
that it is in the nature of information systems to yield 
information even about their own performance. 

Yet, archival data was employed as a primary method in 
fewer than 18% of the reviewed works (and as a secondary 
method in only 13%) . As a non-obtrusive method, archival data 
offer the supreme advantage of having the least biasing or 
distorting effects. 

F. DEFINITIONS OF EFFECTIVENESS 

Table F-6 in Appendix F contains a list of the definition 
of system effectiveness gleaned from each reviewed study. 
These are combined and summarized in Table 7 below. 

The first grouping, corporate scores, represents the 
traditional measures of success in the private sector. Among 
the reviewed studies, they are a minor fragment of the 
definitions employed for information success. The next 
traditional measure of system effectiveness, efficiency (cost 
reduction in this case) was found in 13% of the studies. 
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It is a definition how much under attack from the user 
attitude enthusiasts. 



TABLE 7 

DEFINITIONS OF EFFECTIVENESS FOR INFORMATION SYSTEMS 



Corporate score 




4% 


Profit 


2% 




Return on investment 


2% 




Efficiency 




13% 


Cost reduction 


13% 




Intermediate goods 




29% 


Information usage 


5% 




Plan implementation 


2% 




System usage 


18% 




Job factors 


2% 




Office automation 


2% 




Effectiveness 




9% 


Decision quality 


7% 




Performance factors 


2% 




Value Assessment 




7% 


Value of information 


5% 




System value 


2% 




By-product 




34% 


User attitude 


34% 




Miscellany 




2% 


Improved grammar 


2% 





29% of the studies counted as a system output a factor 
that is actually a system input (described in Table IV-6 as 
an intermediate good) . The mistake in these cases was to see 
the usage of information or a system as evidence of the 
system's success. The correct question is whether results 
from the usage, not the level of usage itself, benefit an 
organization. While there is much discussion in the MIS 
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literature on measurement about the need to focus on 
organizational effectiveness rather than conventional cost- 
cutting efficiency, only 9% of the reviewed studies employed 
effectiveness as a definition of success. 

The major definition of system effectiveness found in 
these studies was some expression of user attitudes. This 
approach holds that if people say they are happy with a system 
then the system is a success. That seems sensible enough. 

Yet, the popular and trade literature reviewed in Chapter 
I found that the payoff from information systems is either 
falling or not advancing commensurate with the investment made 
in it. A user's attitude about a system is an entity quite 
separate from the economic contribution of a system makes to 
an organization. The assumptions in this approach take either 
of these two forms: 

1. A user is satisfied with a system that is productive. 
A user's satisfaction is a' surrogate measure for a 
system's productivity. 

2. Technical efficiency and economic contribution are not 
the point. At best, they are the means. User 
satisfaction is the end. 

In the first case, the possibility exists that a user can be 
satisfied with an unproductive system. The satisfaction of 
a user may not necessarily be based on a system's efficiency 
but on other factors such as job-enlargement, better equipped 
work areas, learning of new skills or protection from 
redundancy . 

In any case, system efficiency and performance must be 
examined apart from the attitude of users. It would, for 
example, be interesting to examine situations in which users 
were highly satisfied with unproductive information systems. 
That interest can be pursued only if satisfaction and system 
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performance remain distinct. The second case represents a 
matter of ideology, most often among the organizational 
development group that has emerged from organizational 
psychology. The root of the user-attitude definition of 
effectiveness may be methodological. 

If a group is committed to the method of asking people, 
then the resulting answers are arrogated to an end. If you 
cannot measure effectiveness at the system level then it is 
expedient to dismiss it as a focus of inquiry. Can a formal 
theory of information system effectiveness be constructed on 
a foundation of attitudes about a system rather than 
measurements directly from a system itself? 

G. REPORTED CHANGE IN SYSTEM EFFECTIVENESS 

Table F-7 in Appendix F lists the level of change in 
organizational effectiveness, following the implementation of 
an information system, reported by each of the 45 empirical 
studies reviewed here. 

In a group of studies that purport in some or way other 
to report findings of an inquiry into the success of 
information system, only 20% actually did so. A serious 
question arises as to whether MIS can construct a formal 
theory of system effectiveness when changes in effectiveness 
are not reported by 80% of its empirical efforts? When MIS 
claims to be studying system effectiveness yet has no findings 
related to it, MIS is in danger of gravitating into 
irrelevancy in the estimate of its major constituents: senior 
line managers, MIS practitioners and the general public. 
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V. CONCLUSIONS 



A. PROBLEMS IN ESTABLISHING CAUSAL INFERENCE 
1 . R**«arch Dasigna 

Much of the concern about credible, appropriate 
research designs for the examination of the effectiveness of 
information systems has been treated in the previous chapter. 
A concern that still merits exercising is the proportion (two 
thirds) of the studies in this review that were of the one- 
shot case design. 

According to Denzin (1970) , the one shot approach 
makes no attempt to approximate the virtues of experimental 
design: no control, no control group, no time dimension. 
Rather, it simulates an experiment-, not through design but, 
through analysis, specifically multivariate analysis. 

This technique creates analytic groupings out of the 
collected data after the fact, groupings that the researcher 
should have been seeking initially. The underlying 
vulnerability of multivariate analysis is that any purported 
causal relationships are tentative due to the lack of a time 
dimension. Nagel concludes (1961) that the difficulties of 
basing causal inference on multivariate analysis "are 
notorious . " 

a. Internal Validity 

73% of the studies reviewed here were vulnerable 
to alternative explanations of their findings because they 
could not control for the effects of history and subject 
maturation in their research designs. 84% were similarly 
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vulnerable to rival explanations of their findings because 
they couldn't control for the effects of selection bias and 
subject mortality in their designs. 

Further, 22% of the studies were vulnerable to an 
interaction effect of selection bias and subject maturation. 
Another 4% of the works had their findings threatened by the 
effects of instrument calibration and re-test sophistication. 
Only four our of the 45 studies controlled all threats to 
internal validity: two true experiments and two posttest-only 
control groups. 91% of the reviewed studies were in a 
situation of having their findings open to one or more 
alternative explanations other than those reported by the 
researchers . 

If this group of empirical examinations of system 
effectiveness cannot state, with confidence, that their 
findings are caused by the variables claimed by the 
researchers, but might well be caused by effects that were 
not controlled by the designs of the studies, then they lack 
a basis for establishing causal inference. They cannot be 
the foundation upon which a formal theory of information 
system effectiveness can be built. 

b. External Validity 

89% of the studies reviewed here were vulnerable 
to the possibility that their findings cannot be generalized 
to a population beyond their sample subjects because of the 
effect of selection bias interacting with the experimental 
variable. Their vulnerability to this effect is inherent in 
their research designs. 11% faced the same problem due to 
the effect of re-test sophistication interacting with the 
experimental variable. Only 4% of these studies were, by 
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virtue of their research designs, able to control for any of 
the threats to external validity. Out of the 45 studies 
reviewed, all were vulnerable to at least one of the threats 
to external validity. 

While external validity does not directly affect 
the establishment of causal inference, it does raise the 
question of the relevancy of these research efforts if they 
do not reflect any segment of the real world beyond their own 
samples. Of how much value in the construction of a formal 
theory of system effectiveness will there be in studies that 
speak for nothing more than their own research arrangements? 

3 . Time Dimension 

Out of the 45 studies reviewed in this thesis, 89% 
have no way to account for time due to the structure of their 
research designs. In the words of Denzin (1968), studies of 
this sort take a "snap shot" at one point in time and leave 
it to the respondent to weave a tale about events preceding 
the interview or questionnaire. 

Causal inference requires that empirical findings be 
able to determine that a variable X preceded a variable Y in 
time in order to support the explanation that variance in X 
caused subsequent variance in Y. MIS assessments of system 
effectiveness lack this critical requirement for causal 
inference. Thereby they also lack another foundation for 
building a formal theory of system effectiveness. 

4 . Data Collection Techniques 

More than two-thirds of the reviewed studies gathered 
data on system effectiveness variables by asking direct 
questions of human respondents, whether by self-completed 
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questionnaire, survey interviews or by phone. Fewer than a 
third used methods other than the direct asking of questions. 

This renders the greatest part of the MIS studies of 
system effectiveness vulnerable to the possibility that the 
data they recorded reflects something other than the data that 
the researchers believe they collected. The direct asking of 
questions is a method that is obtrusive and socially 
demanding . 

Questionnaires and interviews are suspected of 
creating the very data they record rather than recording data 
on events or processes external to the research act. As with 
internal validity, obtrusive methods raise the issue as to 
whether the reported findings of such studies are due to the 
causes claimed by the researchers. 

a. Survey Research Problems 

(1) Penetrating Language of Group Being Studied. 
Reliance on survey methods - whether self-administered 
questionnaires or personal interviews - raises a number of 
problems. First is that the interviewer or the writer of the 
questionnaire must penetrate the language and symbols of the 
group being studied (Denzin, 1970) . Otherwise, the interview 
or the questionnaire are carried out in a relationship in 
which the researcher interacts with the respondents as 
strangers (Denzin, 1968). This is a process readily 
appreciated by anthropologists undertaking to study an alien 
culture. It is a process ignored by academics pursuing the 
study of a group that seems to be less remote. The assumption 
by MIS academics that the languages and meanings used by their 
subjects are already known is a fatal conceit. 



57 



(2) Differential Status of Respondents . A second 
problem - for which there was no evident concern in the 
studies reviewed here - is that different individuals within 
a group occupy different social positions in a group itself. 
Consequently, their perspectives on the group and its success 
or failures will vary by position. The apparent operating 
assumption in these studies was that members of a group are 
essentially interchangeable parts. Each represents the group 
through undifferentiated expressions. This has the effect of 
yielding responses that are actually quite different from what 
the researchers believe they are gathering. 

(3) Obtrusive Nature of Survey in Everyday Life. 
A third problem concerns the interaction between the 
questionnaire or the interviewer and the respondent. The 
operating assumption in the reviewed studies seems to be that 
the questionnaire or the interview is neutral, eliciting a 
straightforward recitation of facts or of opinions not 
influenced by the question-asking act itself. 

In actuality, the opposite is the case. 
Respondents are anything but straightforward with question 
askers. This involves socioeconomic, cultural, ethnic or 
organizational status differences between researchers and 
subjects. Male and female differences is a common source of 
constrained reporting by respondents (Denzin, 1970) . So are 
racial differences. These act to create responses out of the 
interaction itself rather than collect data on a phenomenon 
outside the interaction such as, say, systems effectiveness. 
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(4) Social Definition of the Research Process. A 
fourth problem is that the researchers, the topics, the 
respondents, the questions, the answers, the setting, and even 
the questionnaire form are objects within a research 
interaction. Their meaning is not objectively defined outside 
of, or prior to, the research act. Rather, the social 
definition of these objects takes place as a negotiation 
process within the interaction. This is a process about which 
the researchers in this review showed no awareness. 

Again, the data collected may not provide an 
accurate basis for the findings claimed by the researchers. 
As they are described by Reciting (1966), "interviews and 
questionnaires intrude as a foreign element into the social 
setting they would describe, they create as well as measure 
attitudes." Research methods, as Denzin (1968) argues, are 
not "sterile a theoretical tools." 

(5) Role taking in Survey Responses . A fifth 
problem concerns the credibility of survey responses. As was 
discussed in Chapter IV, people take roles with respect to 
questions. They are vulnerable to Guinea-Pig effects and to 
taking on roles such as antagonist, sympathizer, helper, test- 
taker, informant, protector, participant in a scientific 
endeavor or spokesperson for a group (answering not for 
themselves personally but for an imagined collectivity) . 

\ These roles put a respondent in a frame of mind 
other than that of a straightforward provider of research 
questions. The latter is probably the very role not taken up 
by respondents. Yet, straightforward provider of answers is 
the only role believed to be presented by survey researchers. 
The result is that the data collected by surveys means are 
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something other than what the researchers believe them to be. 
Researchers analyze these data and report them without any 
suggestion that they are affected by the very process of 
collecting the data. 

(6) Social Desirability Effect. A sixth problem is 
social desirability. This takes place within the interaction 
between a respondent and an interviewer or a questionnaire 
(where the responses are not anonymous or not believed to be 
truly anonymous) . In North America, and possibly Western 
Europe, a respondent gives answers that would be socially 
acceptable by a standard known to the respondent (usually 
middle class norms if they are of the middle or lower 
classes) . 

In the United States, respondents usually wish 
to present themselves as happy, mentally healthy, educated and 
prosperous. While interview schedules and questionnaires can 
be constructed to detect and discount socially desirable 
responses, there was no indication that this problem was dealt 
with in any of the reviewed studies . 

(7) Fallacy of Reductionism . A seventh problem 
with the survey method is that it necessarily gathers data 
from individuals. Missing are data about social aggregates 
such as organizations, groups, teams or systems. In that 
these 45 reviewed studies purport in some way to examine the 
effects of information systems, it would seem appropriate that 
the data collection would focus on systems per se . 

What these researchers, did instead, was to 
focus on individuals, thereby indulging the fallacy of 
reductionism: that a functioning whole is the same entity as 
the sum of its component units. This is, in the view of 



60 



Wertheimer (1950), a crucial problem in social research in 
which functional wholes have not been defined as systems the 
behavior of which is not determined by that of their 
individual elements . 

A metaphor suggested by Mill (1879) says that, 
"the different actions of a chemical compound will never be 
found to be the sums of actions of its separate parts." 
According to Nagel (1961) functional wholes are systems which 
cannot be built up out of elements by combining the latter. 
A functional whole cannot be properly analyzed from an 
additive point of view. A whole cannot be inferred from 
properties displayed by its constituents in isolation from the 
whole . 

Coleman (1958) complained that no matter how 
numerous the correlations, how complex the analysis, survey 
studies focus on individuals. They tend to be, in his phrase, 
no more than "aggregate psychology." What these studies miss 
are the relationships between individuals. They show nothing 
about the interactions between people. 

(8) Inability to Measure Behavior . An eighth 
problem with the survey method is that it is inherently unable 
to measure behavior. It can only record attitudes. As 
discussed in the previous chapter, a survey that asks 
questions about behavior can only record a person's attitudes 
about the questioned behavior; a survey cannot show the actual 
behavior. This is because, as Denzin points out (1968), the 
emphasis on "respondent reconstruction" of past events, the 
survey is destined to gather data only on attitudes and not 
behavior . 
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Coleman (1969) described this as a shortcut 
taken by most researchers. The study of behavior requires 
observation (watching, participating or experiencing) or 
recording of behavior (archival records, audio tapes, video 
tapes, films). This requires time. It requires equipment. 
It requires digging through records. It requires theoretical 
sampling (Denzin, 1970) of the places where the behavior will 
take place. The convenient assumption behind attitudinal 
research is that there is a connection between attitudes and 
behavior; to wit, attitudes cause behaviors. 

While there is evidence that behaviors lead to 
the development of attitudes, showing that the causality runs 
from attitude to behavior has been problematic (Denzin, 1968) . 
On the contrary, evidence has accumulated showing that 
accurate prediction of behavior from measures of attitudes is 
not possible with the methods now employed (Tittle and Hill, 

1967) . 

Studying behavior takes a greater commitment to 
scientific endeavor than is typically required for a one-shot 
design, mail-out questionnaire. Therefore, what is observed 
in these 45 studies, and throughout the general practice of 
research in management topics, is a gravitation to expediency: 
the convenience, the speed, the low cost of the survey method. 

In summary, the survey fails to meet even the 
minimal assumptions for an adequate methodology (Denzin, 

1968) . 
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b. Questionnaire Problems 



(1) Fixity of Questions . In addition to the 
preceding problems of surveys generally that afflict 
questionnaires as well as interviews, questionnaire schedules 
have additional problems peculiar to their usage. The 
reviewed studies that employed questionnaires reported that 
they were of the standardized schedule type. That is, 
respondents were presented with a fixed set of questions and 
a preconceived, precoded, finite set of response 
possibilities . 

The result of this rigid approach was to 
generate numbers, through the questionnaire process, about 
what was on a researchers mind when the questions were 
written. It is a process that taps little of what is on 
respondents' minds. The likelihood is that the response to 
the indifference of questionnaire fixity is simply fabrication 
(Denzin, 1970) . 

(2) Construction of Scales. A second problem that 
is endemic to the questionnaire is the matter of constructing 
scales. The assumption behind the construction of scales on 
attitudinal questionnaires, such as Likert-type instruments, 
is, as Denzin explains it (1970), "that an underlying attitude 
variable is being measured. The scores for any respondent are 
assumed to represent his position on the latent dimension and 
the items are assumed to reflect that dimension." 

Whether the assumption is upheld empirically is 
problematic for all questionnaires using scaled responses for 
indications of attitudes or opinions. The difficulty is that 
no single indicator can cover all dimensions of a variable 
(Denzin 1970) . A single indicator has only a probability 
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relation to what it is that a researcher actually seeks to 
know . 

(3) Mail Out - Mail Back Problems . A problem that 
plagues the mail out-mail back self-completed questionnaire 
is that nothing is known about the characteristics of the 
people who did not return a questionnaire. This leaves this 
technique vulnerable to the internal validity problem of bias 
in the selection of respondents. A face-to-face encounter 
between question asker and respondent can, and often does, 
involve a "break off" by a respondent. That is, a respondent 
suddenly declines to participate. 

The question asker in these situations at least 
has the possibility of gathering some indications of a 
refuser's age, sex, race or other obvious characteristics. 
Not so the researcher using mail back responses. The chief 
virtue of the mail out-mail back technique is that is cheap 
and easy. It is not a means for gathering data that will 
contribute to the building of formal theory. 

5. Definitions of System Effectiveness 

The definitions of system effectiveness gleaned from 
these 45 studies tell a great deal about what is taking place 
in the endeavors of academic MIS to measure the performance 
of systems. The field has polarized into two schools of 
thought: cost-benefit analysis and user attitudes. 

Actually, it is less a case of polarization than it is 
of the aggressive user-attitude camp driving cost-benefit into 
retreat. The user-attitude enthusiasts view do not care if 
their definition of system effectiveness as a resultant of 
scientific endeavor. For them it is a matter of ideology. 
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The user-attitudinalists characterize the use cost- 
benefit analysis as discredited, traditional and infeasible. 
They view user attitudes as modern, feasible and appropriate. 
Their argument, to recall the discussion in the previous 
chapter, is that while data can be gathered on costs, nothing 
can be gathered about benefits. Benefits of information 
systems are so lacking in tangibility as to be rendered 
useless as a way to analyze effectiveness. 

Cost-benefit analysis was appropriate for massive, 
batch-processed transactions on mainframe systems. They are 
part these measures of the cost-reduction efficiency 
mentality. What value can have, the attitudinalists ask, when 
we are examining the effectiveness of, say, a microcomputer- 
based decision support system? 

In the latter case, the emphasis is not upon 
efficiency but upon effectiveness. It is not upon quantity 
and cost but upon quality. Therefore, the very advancement 
of hardware technology and the evolution of software systems 
have rendered cost-benefit analysis as deficient and 
inappropriate . 

The only remaining issue, in the minds of the 
attitudinalists, is which particular attitudinal 
questionnaire, and which specific question items, are best 
suited to gauge user satisfaction. That, indeed, is the focus 
of therr research endeavors. Not the construction of a formal 
theory of system effectiveness but the fine-tuning of the 
ultimate questionnaire. 

Left out of this approach are any indicators of the 
actual productivity that arises from a system where efficiency 
is the issue or the actual effectiveness of a new system where 
that is the issue. Missing are measures of actual 
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productivity as distinct from users' opinions about 
productivity . 

As the discussion of this issue in the last chapter 
indicated, there is good reason to believe that the conceits 
of the attitudinalists about the superiority of their 
formulation of system success is, in reality. A fig leave 
covering their methodological nakedness. As graduate 
students, one suspects, they invested in learning the 
prevailing but deficient survey method employing standardized 
Likert-scaled questionnaires in mail out techniques operating 
under one-shot case designs. 

The opportunity cost of having learned that approach 
has been an ignorance of other approaches. The 
attitudinalists are not only ignorant of alternative 
approaches, they are ignorant of the problems and deficiencies 
of their own methods. 

These suspicions about, and characterizations, of the 
user-attitudinalists would be no more than material for 
empirical testing and academic contention were it not for 
their own aggression. Not content to demonstrate that their 
way might be better, they attack alternative concepts of the 
effectiveness of information systems. They force 
practitioners of cost-benefit analysis into a defensive 
posture . 

At the very least, they make difficult an attempt to 
reconcile different views of effectiveness. If the concept 
of efficiency cannot be entered into the dominant dialogue 
about system performance, it cannot be compared or combined 
with user attitudes. 
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B . INTELLECTUAL FOUNDATIONS OF MIS 



1 . Failure to Establish Causal Infaranca 

The choice of research designs, the vulnerability to 
threats to internal validity, the lack of accounting for a 
time dimension and the choice of data collection methods - 
considered separately and taken together - in the MIS 
empirical assessments of information system effectiveness 
provide no grounds for establishing causal inference. 

2 . Inability to Build Formal Thaory 

The inability to establish causal inference in MIS 
studies of systems removes a foundation for building a formal 
theory of system effectiveness. By derivation, it further 
removes one basis for constructing a formal theory of 
information systems. Not to construct theory is to miss the 
point of academic endeavor. In the words of Nagel (1961), 
"the paramount aim of theoretical social science is to 
establish general laws which can serve as instruments for 
systematic explanation and dependable prediction." 

3. A Discipline without Theory 

MIS is an academic discipline that cannot construct 
formal theory about its essential concerns. Its has put 
itself in the position of being unable to adequately describe, 
explain or predict the very matters that justify its 
existence. It has traded the pursuit of explanation and 
prediction for the pursuit of career security. Publication 
is pursued as a means to tenure rather than as a means to 
supporting MIS practitioners. A gulf will widen between the 
academic world of MIS and the practitioner world of MIS. 
Thereby, does academic MIS flirt with irrelevancy. 
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VZ . RECOMMENDATIONS 



A. FOR MANAGERS AND COMMANDERS 

1 . Th« Cost of What Is Not Known About Inf onnation 

Systems 

The United States Navy has hundreds of thousands of 
desk-top microcomputers. The very number provokes a concern 
as to how efficiently or effectively they are being used, a 
concern that emanates from high levels in the Navy. Read 
Admiral Paul Tobin, the U.S. Navy's chief of data automation 
systems, wondered aloud recently how many of the Navy's 
desktop machines were being used. He was quoted in Federal 
Computer Week as saying, "Let's make sure we're using our 
tools. In some way the system has gotten ahead of the users. 
[We] have to figure out how to do more [with the basic 
systems]" (Brewin, 1989). 

Senior military officials, much like senior corporate 
executives, encounter difficulties in dealing with the world 
of information technology. Senior officials often have little 
grasp of the technology of computers. They know that 

information systems can be used to strategic advantage but 
they are at a loss to transform their wishes into successful 
implementations. Therefore some officials delay information 
technology decisions. Others avoid them altogether. Still 
others take the opposite tack and blindly provide a blank 
check for the purchase of information systems, especially 
microcomputer-based systems, without a clear expectation of 



68 



the return they expect, the results they want or how to 
measure the effect of what they getting. 

Admiral Tobin has summarized what most observers of 
the approach by the U.S. military to the new information 
technology already know: money has been thrown at hardware and 
software in the past decade without a sense of what was wanted 
or of what was achieved. That is not the same thing as saying 
that the U.S. military has received no return for its 
expenditures. Only that (1) it is unknown what was achieved 
and that (2) consequent achievements will have been the result 
of good fortune and the dedication of field level activity 
system-enthusiasts rather than as the result of planning. 

A good deal of our alleged low productivity and low 
payoff from information systems, especially office automation, 
is due in large part precisely to the generous but 
unscrutinized funding for microcomputer-based information 
technology. Millions have been spent on computers, 
peripherals, supplies, environments and training without the 
hard review that would be given to a turret lathe costing only 
thousands. A lack of scrutiny before the office automation 
funding frenzy has generated more managerial remorse than 
office productivity. In hindsight, managers and commanders 
are asking, "Where is the computer payoff?" 

This discussion assumes, of course, that senior 
officials truly seek to measure system performance to find 
their payoff. It would be understandable if some officials 
did not want to discover the truth about the productivity of 
office automation. After all, if you have lavished funds on 
desk top computing or information centers, the last thing you 
want to do well or accurately is a study of exactly how much 
productivity was or was not thereby enhanced. 
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2 . Becoming Informed Consumers of Information Technology 

In the world of MIS, there exist a division of labor 
between practicing managers and MIS academics. Managers need 
academic MIS to develop scientifically credible ways to 
describe, explain and predict the working of MIS. Managers 
can't do. It is the job of academic MIS. However, it won't 
get done if managers indulge the inadequate research methods 
of academic MIS. 

Managers and commanders must re-acquaint themselves 
with the requirements for credible research. They must be 
science literate. They must become informed consumers of 
academic MIS studies of such matters as system effectiveness. 
They must be able to know when reported research results are 
scientifically sound. They must beware of the easy dismissal 
of cost benefit analysis and economic analysis by MIS 
researchers from the user attitude school of effectiveness. 
Managers must be clear about the distinction between an 
effective system and people's perceptions of an effective 
system. 

Specifically, managers, as consumers of research must 
beware of studies that purport to assess system effectiveness 
by using attitude questionnaires to gauge system outputs as 
well as system inputs. Attitude questionnaires are 
consistent; they exhibit high correlation coefficients within 
their own domains. The trouble is in relating attitudinal 
scores to independent, external grounding in behavior. 

Where MIS researchers deny that data can be gathered 
on system benefits (too intangible they say) , it is not only 
possible to gather such data, it is relatively easy. 

A further burden that falls to managerial consumers 
of MIS research is to grasp the place of theory. Managers 
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must not let their own action-oriented, practical mindsets 
mislead them into an impatience with the critical need for 
formal theories in MIS. Managers and commanders must ask 
whether the research results they read are part of an effort 
to build theories that can describe, explain and predict MIS 
system effectiveness. If not, what are the researchers trying 
to do? Are they simply doing studies to gain academic tenure? 

If managers sense that academic MIS, as represented 
by research reports in academic MIS publications, have no 
relevance to their work, they must make that clear to the MIS 
academics. Unfortunately, managerial MIS does not pay for 
academic MIS directly. The income of academic MIS is not 
dependent upon the pleasure of practicing MIS. This structure 
of third party financing of research allows MIS academics to 
be distracted from the construction of theory into the 
expedient pursuit of careers. 

However, MIS practitioners do provide academic MIS 
with its legitimation, its reason for existing. Legitimation 
is not a strong control in the short run. In the long run, 
without acceptance by practicing MIS, academic MIS is 
irrelevant . 

This situation is different in the interface between 
MIS managers in the defense community and academic MIS 
researchers attached to defense educational institutions. 
There, 'the practitioners provide not only legitimation but 
also research funding. 

MIS academics in the defense community are held 
accountable for their research results. While this control 
possibility is not perfectly exercised, its potential is large 
for disciplining academic MIS research. 
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B. FOR THE STUDY OF SYSTEM EFFECTIVENESS 



1 . Research Designs 

a. Time Dimension 

Until MIS research designs for studying system 
effectiveness account for the time dimension, they will lack 
any basis for causal inference (Vitalari, 1985) . All MIS 
research designs must gather data at more than a single point 
in time. The portion of the MIS studies that have no concern 
with time in system assessment (now 89%) must be reduced below 
25% in the next decade. Otherwise, MIS research may fail to 
establish scientific credibility. MIS researchers must 
abandon cross-sectional analysis designs in favor of 
longitudinal designs. 

b. Randomized Comparison Groups 

As Nagel (1961) noted, it is possible to advance 
scientifically without meeting the full requirements of a true 
experiment, yet some form of controlled investigation "appears 
to be indispensable." To that end, MIS researchers must not 
only employ comparison groups in research designs (which 27% 
of the reviewed studies did) but use comparison groups with 
randomized assignment of respondents. 

Only by that means can MIS research control for the 
threats to internal validity arising from history, subject 
maturation, re-test sophistication, instrument calibration, 
statistical regression, selection bias, respondent mortality 
and the interaction of selection bias and subject maturation. 
Only then can MIS research build a confidence that its 
findings actually reflect what MIS researchers intend instead 
of being open to rival explanations by effects which were not 
taken account of in the research designs . 
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c. Qusai -Experimental Designs 

While a great deal can be learned from laboratory 
experiments, the rich and fruitful MIS research of the future 
will be done in field settings. Most of it already is (96% 
of the reviewed studies were done in field settings) . The 
difficulty is that the experimental virtues of repeated 
measures with randomized controls are not now being 
approximated . 

The route to gaining the richness of field setting 
research with the rigor of experimental controls is quasi- 
experimental research designs (Campbell and Stanley, 1966) . 
MIS researchers must first declare the experimental variable 
they are examining. They must then gather data before the 
application of an X variable and again after . They should 
make longitudinal (multiple observations) , especially in the 
posttest phase. This is not a difficult requirement. It can 
be accomplished with survey methods as well as with 
unobtrusive methods. The beauty of this approach is that 
information systems inherently produce data on key variables 
that a researcher into system effectiveness will want to 
measure . 

To compare organizational functioning in a manual 
work mode with that in a post-implementation mode, researchers 
must commit to digging through accounting and other 
organizational, hard-copy archives. In the post- 
implementation phase, they can then indulge the relative 
luxury of having the system being studied provide the data. 

For an example of this technique see the one-group 
pretest/posttest study of a standardized contracting and 
purchasing system by Barclift and Linson (1988) . 
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2 . Triangulation 

Triangulation is amply illustrated by the proverb of 
the blind men reporting on the characteristics of an elephant. 
Each man reported different characteristics, depending on 
whether he was making a tactile examination of the trunk, 
tusks, legs or tail. The situation is much the same in MIS 
research. Each researcher, employing a single method of data 
collection, a non-experimental design and a single theoretical 
position on the nature of system effectiveness is little 
better than a blind man. Only conceits and psychic 
investments in current methods keep MIS researchers from 
appreciating the limits of their empirical insights. 

When the proverbial blind men can compare, contrast 
and locate their findings, they open the potential to 
describing the illusive reality of the elephant. Once MIS 
researchers use multiple methods of data collection, multiple 
observation points in time, multiple comparison groups and 
multiple forms of data then they, too, will be open to the 
rounded, realistic description of the demonstrated (rather 
than the assumed) effectiveness of information systems. 

a. Triangulation of Data 

(1) Triangulation of Time. Of the reviewed 
studies, specification of time was inadequate. The most 
specific mention was the year in which data were collected. 
Beyond that, no attention was given to the timing of data 
collection. Triangulation of time would require data 
collection with these variations: different times of year, 
different times within a month (a great deal of information 
system activity has a monthly cycle) , different days of a week 
and different times during a day (this is a potent but 
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neglected avenue of research into, for example, the way a 
night shift crew uses a hospital information system without 
the support of daytime MIS staff) . 

Not all of these variations may be relevant to 
any particular examination of an information system. However, 
the likelihood is that one of them will profoundly affect the 
data gathered from any one type of information system. 

(2) Triangulation of Place. All human activity has 
a spatial designation. Many of the reviewed studies noted 
the organizational location of their data collection that 
often implied a geographical variation. Yet, spatial 
differences were not introduced consciously as a variable. 
Space differences are attended to cross-cultural comparisons, 
but cross-cultural comparisons have not been a part of system 
effectiveness inquiries. 

The temptation in MIS research is to assume 
that place is irrelevant without further consideration. The 
conceit is that influence of information technology overcomes 
differences in time, culture or place. But does it? Do 
studies exist that establish that spatial location can be 
excluded as a critical independent variable? No. Instead, 
place is conveniently assumed away: information is 
information . 

Yet, information is an object in our lives 
subject to the same, situated negotiations among interacting 
people as any other object (Boland, 1985) . The influence of 
spatial location is an empirical question; until its effect 
is deliberately examined, it cannot be assumed away. 
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(3) Triangulation of Level of Data. This thesis 
has already exclaimed upon the need to move MIS studies on 
system effectiveness away from data collected from individuals 
to data collected about systems and organizations. A 
triangulation of data would require that any study of systems 
performance include data from three levels: (a) individuals 
(which MIS studies now gather in abundance and to the 
exclusion of other levels of data), (b) systems (departments, 
teams, shifts, and true systems of people who are connected 
in relationships that are built around producing an output 
through the mediation of hardware and software of their 
information technology no matter what their formal 
organizational identities) and (c) interactions (the 
encounters, negotiations, definitions, avoidances, 
competitions, cooperations, and coordinations between 
individuals or groups but not characteristics of the people 
or the groups per se ) . 

Jb. Triangulation of Thaorias 

To speak of triangulating theoretical explanations 
of system effectiveness is premature in a discipline that has 
not developed a formal theory nor is presently able to do so. 
Yet, we have emerging, competing schools of thought as to how 
the concept of system effectiveness should be defined. Any 
study of system effectiveness ought, therefore, to approach 
an empirical assessment with more than one concept of 
effectiveness . 

A system's performance must be studied in terms of 
user attitudes . At the same time it must be studied in terms 
of costs and benefits. Further, it must be studied in terms 
of the economic value of information. 



76 



And, finally (without exhausting the MIS inventory 
of concepts of system effectiveness), it must be studied in 
terms of organizational effectiveness. 

c. Tri angulation of Methods 

Survey researchers already employ a variant of 
methodological triangulation - within-method triangulation 
(Denzin, 1970) . They use various scales and multiple 
standardized schedules to tap attitudes they are examining. 
The limit here is that this takes place within a single 
method. This leads researchers to congratulate themselves on 
the breadth of their methodological labors while deluding 
themselves that variations on the same method produce distinct 
viewpoints. According to Webb (1966), each method is 
imprisoned within its own biases and distortions. Combining 
within-method variations is a first step but it cannot be the 
only step. 

What is required of MIS research is the combination 
of methods to study the same system or the same phenomenon. 
Thereby will the strengths of one method compensate for the 
weaknesses of another. The effectiveness of a information 
system implementation can easily be assessed using unobtrusive 
measures as well as, say, attitudinal questionnaires as 
Barclift and Linson (1988) did. 

The greatest benefit of between-method 
triangu'lation is countering rival explanations for reported 
findings that arise from reactive arrangements. The reactive 
effects of questionnaires and interviews on data were 
discussed in the previous two chapters. In an ideal strategy, 
MIS researchers will combine survey questionnaires, personal 
interviews, archival data, system-generated measures. 
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participant observations and phenomenological techniques 
(Webb, 1966/ Denzin, 1970/ Boland, 1985) . 

3 . Long Term Field Studies 

Even with a recommendation for accounting for a time 
dimension and for triangulating for time as a data variable, 
the preceding discussion has been framed in the assumption of 
a relatively short term research project measured from months 
to as long as two years. Covering the implementation of an 
information system, from system proposal to final, post- 
training full implementation could easily cover two to three 
years. But MIS research must commit to going beyond that. 

Once the introduction of a system has been assessed, 
MIS researchers must continue with follow-up studies through 
the life cycle of a system. Considering that the useful life 
cycle of some information systems is quite short (most are 
shorter than a decade and some scarcely last three years 
before succumbing to obsolescence), this is not a burden. 
Once entry has been made to an organization to study its 
information systems, the greatest hurdle is past. Unless MIS 
researchers have a propensity to be wearisome to their hosts 
and wear out their welcomes, they must extend their research 
relationships for a long view. 

The payoff form this is that different things happen 
to information system in the long term than are ever imagined 
in the short term. The most potent revelations in social 
research have come from the longest studies: communities, 
organizations, cohorts of people over a lifetime. The same 
will be true with MIS topics. In their examination of the 
introduction of a productivity-enhancing system that cut 
contact-processing time in a U.S. Army installation, Linson 
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and Barclift (1988) suggested that follow-on research was 
needed to see if the short term efficiency, in itself, would 
provoked user activities to make greater demands on the system 
in the years ahead. 

4 . Onobtru»iv« H«uur«i 

A theme that has been exercised in the fourth and 
fifth chapters of this thesis, and echoed earlier in this 
chapter, is the distorting, biasing effects of the survey 
method. This leads to a recommendation for MIS research to 
embrace the methodological point of Webb at al (1966) , for 
the use of data collection techniques that are not reactive 
- that is, that do not lead to role-playing, fabrication, 
social desireable response and other behaviors that arise in 
the research interaction. 

This recommendation is for the use of nonreactive data 
sources, specifically archival data sources. This 
recommendation is particularly appropriate to MIS research 
because the production of information is the point of 
information systems. Some of that information reflects the 
performance of the system itself. This does not mean the 
technological monitoring of hardware system in terms of 
processor clock speed or input/output efficiency. Rather, it 
means the use of organizational accounting data on costs, 
production and outputs. 

-An argument will arise from the user-attitudinalists 
that the archival data produced by information systems will 
miss the intangible benefits of an information system such as 
higher morale. Once again this reflects an investment in a 
single method rather than the reality of available data. 
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One approach to measuring morale is to use attitude 
questionnaires that purport to assess it. But only attitudes 
will be assessed. Do the attitudes inform the behavior of 
people? Does it direct what they actually do? As an example 
of an alternative measure of morale, examine organization 
records for the classic workplace-safety-valve for people who 
are unhappy with their jobs: sick leave (Barclift and Linson, 
1966) . When people are truly are fed up with their work, 
especially people below the managerial or professional levels, 
they avoid the workplace by staying home and calling in sick. 
Or, they drive themselves into actual symptoms of illness 
produced from anxiety and job stress. Why then must an 
attitude questionnaire by employed exclusively to assess 
morale when a powerful, triangulating behavioral measure is 
so available? 
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studies are needed to test the relationship between 
satisfaction and bottom line indicators of user and 
organization performance. 
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information needed for the management decision-making 
process that will be used if it is made available to 
either private or public sector managers. 
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"(Use) of a system is one of the " key " variables 
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otherwise non-operationaL. 
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Spring 1988. 

A questionnaire's validity is the extent to which the 
questionnaire actually captures the concept it purports 
to measure. 

The first role of the UIS measure is to detect the 
presence of a problem with user satisfaction in an 
organization. This would be achieved by administering a 
UIS survey and examining the total satisfaction scores. 

It is critical that the interpretation of UIS results be 
treated cautiously and contextually. 

The UIS measure provides a way to obtain information about 
overall user satisfaction with information services as 
well as more focused information on specific satisfactions 
in the three areas of EDP staff and services, information 
product, and user knowledge and involvement. 

The real value of a UIS measures lies in its ability to 
discriminate among a large number of possible problem 
areas . The situation and the purpose of the study should 
guide whether the manager/researcher choose the long form, 
the short form, or a simple one-question UIS measure. 
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Information Systems 
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July 1984. 

Post implementation evaluation of computer-based 
information systems is of vital importance for improving 
the operation of current systems and in the design of new 
applications . 

Three major activities in which this plays an important 
part : 

1. The acquisition of equipment. 

2. In improving the operation of production systems. 

3. In the design of new applications; this is extremely 
important, both to the computer staff and users of the 
system. 

A cost-benefit study is one of the best ways to measure 
it. Obviously, the benefits from a computer-based system 
should exceed the cost of developing and running it. 
Unfortunately, assessing the benefits has been difficult, 
particularly for systems supporting sophisticated 
decisions; i.e., beyond the routine processing of 
transactions . 

Cost-benefit is best but too difficult - so use user 
satisfaction. Use and satisfaction are relative, that is 
they are measure on a continuum, not a binary scale. 



Colton, Kent W. 

Computer and Police: Pattern of Success and Failure. 
Sloan Management Review , vol.14, no. 2, pp. 75-98. 

Winter 1972-73. 

Inventory of problems, not a measure of success. No 

metric of effectiveness, only impressions. 

There are indications that computer use may lead to 
centralization of power. Of those interviewed during the 
on site visits, 96% felt that computer use would lead to 
better information and grater control of the police 
department by those at the middle and top management 
level . 

It should be remembered that the field of computer 
technology is still in its infancy. Perfection should not 
be expected instantly in an area so young and rapidly 
changing. On the other hand, there is a certain mystique, 
as well as commercial force, surrounding the application 
of computers. This has led to high expectations and in 
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some respects to oversell. As a consequence, there is a 
need for a continuing assessment of the benefits and 
impact of computer use. 



Cron, William L. and Marion G. Sobol 

The Relationship Between Computerization and Performance: 
A Strategy for Maximizing the Economic Benefits of 
Computerization 

Information and Management , vol.6, no. 3, pp. 171-181. 

June 1983. 

Surprisingly the immense growth in business usage of 
computer facilities and software in the past decade has 
not been accompanied by much analysis of the influence of 
computer ownership and compute usage on the sales, costs, 
and profitability of the firm. Part of the problem is one 
of cause and effect: whether higher computer usage leads 
to more profit or whether higher levels of profit lead to 
more computer usage. A second difficulty is that of 
estimating the sales or profits if computer systems had 
not been introduced. 

They never discussed how performance should be measured 
or related to these variables. 

Other attempts at evaluating performance of information 
systems have relied on management opinions, based on user 
surveys. Thus Neumann and Segev state "a universal 
measure like net effect on earning cannot be applied yet". 

In summary, the literature shows that relatively few 
studies of these effects have been made. 

Result of this study indicate that computerization is 
positively related to the overall performance of medium 
and large size firms. Small firms should be cautious in 
estimating the potential for significant economies of 
scale from computerization, but should use information to 
facilitate management control and decision-making. 



Decker, Jane E. and John P. Plumlee 

Microcomputers, Manpower and Performance in a Public 
Utility 

Public Productivity Review , vol.9, no. 2-3, pp. 213-225. 
Summer/Fall 1985. 

Rarely, if ever, can managers simply deploy microcomputers 
into the organization and receive a clear-cut productivity 
bonus . 
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The Performance Improvement. Division sought and acquired 
microcomputers not to create a system but to enhance its 
own productivity within that system. 

The JEA experience demonstrates dramatically that the new 
technology of microcomputers provides real opportunity for 
the enhancement of public sector productivity. 



Durand, Douglas E., Rex 0. Bennet and Samuel Betty 

What Does Information technology "Do" to Business 
Communications?: Two Empirical Studies 
Information and Management , vol.13, no. 3, pp. 111-117. 
October 1987. 

Yet surprisingly little empirical research has been 
conducted to determine the effect of information 
technology on business communications. 

Productivity can be divided into two segments: efficiency 
and effectiveness. Both studies suggest that end-user 
word processing communications result in different output 
than those produced by conventional means. 
Organizationally, productivity will decline unless the 
communication is more effective. Does the write's 
communication style effect the manager's confidence in 
decisions? Thus, important questions remain regarding the 
effectiveness of communications technology. 



Edelman, Franz. 

Managers, Computer System, and Productivity 
MIS Quarterly , vol.5, no. 3, pp.1-19. 

September 1981. 

One measure of system's effectiveness is the degree of its 
organizational impact. 

The best load to more efficient office operations is 
through more effective support the managers. 

Each dollar invested at the top of the pyramid in 
effective system technology can be expected to replace $X 
(X dollar) in white collar labor cost. 
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Edstrom, Anders., 

User Influence and the Success of MIS Projects: A 

Contingency Approach. 

Human Relations , vol.30, no. 7, pp. 589-607. 

1977. 

There are several reasons why one has to resort perceptual 
data : 

1 . Generally one can estimate the cost for developing 
a system but not the benefits, mainly because a 
large number of these are intangible. 

2. Managers do not seem to keep track of these data and 
hence they are not available for research purposes. 

3. Even if we should know the complete cost-benefit 
picture, it still would be impossible to judge the 
relative success with respect to alternative 
solutions to the problem in any objective sense. 

The information needs of the manager are thus crucial for 
the design of the system, and his satisfaction with the 
outcome is certainly a very important criterion of the 
success of the system. 

The perceived success of an information system is, 
however, not only possible criterion of success and it 
seems to be useful to investigate alternative aspects of 
the success. 

The phase of the development process is a further 
contingent factor and that it is important to distinguish 
between types of actors in the development process. 



Englander, Valerie and Fred Englander 

Cost-Effectiveness of Computer-Based Welfare Detection in 
New Jersey 

Public Productivity Review , vol.9, no. 2-3, pp. 271-282. 
Summer/Fall 1988. 

The New Jersey program. . . entails a quarterly computer 
match of the social security numbers of welfare 
recipients. . . against the social security numbers of those 
on public and private payrolls. 

Given that the fraud detection system was implemented 
simultaneously with major national welfare reform 
legislation, the New Jersey program was assessed by 
examining patterns in N.J. caseloads and average grant 
levels relative to the national trends. This methods 
generated an estimated savings of $45.6 million, 
substantially greater than the $6.2 million cost of 
operating the program. This is an encouraging result to 
those policy makers who have recently sought to extend 
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computer-based approaches to detect and prevent fraud in 
other Federal programs. 



Epstein, Barry J. and William R. King 

An Experimental Study of the Value of Information 
Omega , vol.10, no. 3, pp. 249-258. 

1982. 

Virtually every controllable aspect of any information 
system, its design and implementation, are affected by how 
information is valued. 

No satisfactory measure has been developed that has 
general applicability and the question of which measure 
should be applied in a given circumstance has not been 
adequately addressed. 

Most measures of the value of information apply to a unit 
that is at a low level of aggregation. If this is the 
sort of information value concept that must be used in 
designing an IS, the designer is left either to design 
systems idiosyncratically for individual, or to aggregate 
information values across individual in order to construct 
a single measure that best reflects the organization's 
information values. 



Franz, Charles R. and Daniel Ruby 

Organizational Context, User Involvement, and The 

usefulness of Information Systems 

Decision Sciences , vol.17, no. 3, pp. 329-355. 

Summer 1986. 

Two factors reflecting an MIS department's maturity (size 
and age) were found to reduce users' perceptions of system 
usefulness . 

The role of organizational factors was more complex than 
first imagined. Failure of the second canonical 

correlation to produce significance led to the rejection 
of all our hypotheses about direct associations between 
organizational context and user involvement. 

Suggested to strengthen the positive relationship between 
involvement during design and perceived usefulness. 
Paradoxically, MIS departments with broader scopes seemed 
to weaken this same relationship. 
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Fuerst, William L. and Paul H. Cheney 

Concepts, Theory, and Techniques: Factors Affecting The 
Perceived Utilization of Computer-Based DSS in the Oil 
Industry 

Decision Sciences , vol.13, pp. 554-569 
1982. 

Perceived usage, for the purpose of this study, was 
defined as the decision makers' perception of how 
frequently they interacted with the DSS and how important 
the information obtained from the DSS was in assisting 
them to make decision. 

In terms of general use, the factors of user training 
during the implementation process and the accuracy of 
output were found to be important. 

The only factor of importance from the characteristics of 
the decision maker was the number of years of experience 
in the present position, and it was only important for 
specific DSS use. Thus the greater their years of 
experience in their position, the more managers will use 
the DSS for specific, personally initiated reports. 

The important factors in the characteristics of the DSS 
are accuracy and relevancy of output. 

Training and accuracy were found to be important in 
affecting general DSS use; and experience in the present 
position, user training, accuracy of output, and relevancy 
of output were important in affecting specific DSS use. 
Thus, the system's professional should stress those 
factors when designing, building, and implementing a DSS 
in order to maximize the degree to which decision makers 
use the system. 



Gallagher, Charles A. , 

Perceptions of the Value of a Management Information 
System . 

Academy of Management Journal , vol.17, no.l, pp. 46-55. 
March 1974. 

Difficulty measure benefits . A dollar comparison of costs 
with benefits cannot be made. 

This study included one test of a new methodology. The 
result of a single test cannot establish the validity of 
a methodology. The study results suggest that the 
technique is basically useful in this application, nut 
better scales need to be found. 

The finding of this study that managers in certain upper- 
level managerial positions valued EAB reports more highly 
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than other managers suggest the possibility of a general 
relationship between managerial position and the value of 
information. Such a general relationship, if it exists, 
should vary with the type of information and the type or 
organization structure. 



Gallupe Brent R. and Gerardine DeSanctics 

Computer-Based Support for Group Problem-Finding: An 

Experimental Investigation 
MIS Quarterly , pp. 277-296. 

June 1988. 

Two versions of the experimental task, one higher in 
difficulty and the other lower in difficulty, were 
administered to GDSS-supported and nonsupported decision- 
making group, yielding a 2 x 2 factorial design. 

Groups with the higher difficulty task were less satisfied 
than those with the lower difficulty task; however, this 
difference was not significant. The anticipated 

interaction effect between GDSS support and task 
difficulty was not observed. 



The only meaningful finding for this variable was that 
GDSS support resulted in a significantly lower level of 
satisfaction with the decision making process. 

As controlled experimental study the findings cannot 
necessarily be generalized to all levels or types of GDSS 
software or to actual organizational use of GDSS 
technology . 



Hall, P. G., and T. J. Lincoln 

A Cost Effectiveness Study of Production and Engineering 
System within a tightly integrated Manufacturing 
Environment . 

Management Datamatics , vol.5, no. 6, pp. 262-274. 

1976. 

SESAME was primarily established to evaluate the cost 
effectiveness of the computer systems at BAC-CAD . Any 
cost effectiveness study assumes a comparison of one 
situation with another. Although the most straightforward 
method of comparing costs and benefits would have been to 
compare the current computer systems with the cost of the 
number of people required to do the same job, it was felt 
that this would be a meaningless exercise. 
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These considerations suggest that the cost and benefits 
of the computer systems are best compared with the 
realistic manual systems that would probably be introduces 
if the computer was not available. 

The basic philosophy of a SESAME study is that information 
for management is, in our ever more complex environment, 
rapidly becoming one of the most important of company 
resources and that good management demands the most 
effective utilization of the information resource that can 
be professionally obtained. 



Hogue, Jack T. and Hugh J. Watson 

An Examination of Decision-Makers' Utilization of 
Decision Support System Output 

Information and Management , vol.8, no. 4, pp. 205-212. 
August 1985. 

Decisions are considered to include three major phases - 
intelligence, design, and choice. In order for a DSS to 
support the intelligent phase, it must provide information 
of a descriptive nature. 

The primary justification for the development of a DSS is 
that it will be of value to the decision maker. 

Monetary assessment of success is probably not likely 
since it is rare for a DSS user to formally evaluate the 
impact of the DSS on profits. It may be that the use of 
a DSS constitutes a successful system. This study has 
suggested some combination of satisfaction, acceptance, 
and frequency of use as potential indicators of success. 



Kapsales, Peter 

Before and After WP : An Office Automation Productivity 
Study 

Journal of System Management , pp.7-9. 

April 1986. 

If I buy $70, 000-worth of word processing (WP) equipment, 
my operators must still type the names and addresses. How 
much productivity is gained by word processing, when the 
number of key strokes is not reduced? 

Managers must decide what the potential gain in 
productivity is, and if it is large enough to justify the 
cost of the new equipment. It is a difficult decision 
when there is not a reduction of keystrokes. 
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Is Word Processing Productive? 

WP equipment is justified when it: 

a- Saves the typist from making unnecessary keystrokes. 

b. Speeds up and simplifies the document formatting 
activities . 

c. Provides significant benefits by generating letters 
which are personally addressed to customers. 

Automating an office almost always provides productivity 
improvements, but careful analysis should be performed to 
make sure the benefits outweigh the costs. The manual 
environment and the impact automation will have on it must 
be carefully evaluated. 



King, William R. and Barry J. Epstein 

Assessing Information Value: An Experimental Study 
Decision Sciences , vol.14, no.l, pp. 34-45. 

January 1983. 

Comparing opinions of total system with a linear model of 
opinions about separate attributes of system. 

Big error : Calls a scale internal after explicitly 

directing respondents to treat it as ordinal. 

This study demonstrates the practicality of using a 
multiattribute approach to the evaluation of information 
systems. These evaluations were compared with this 
calculated from a multiattribute linear value model. The 
two evaluations were found to be highly correlated. 

The important notion of considering alternative system 
designs is often not operationally applied because of the 
complexity and cost involved in describing alternative 
systems that are directed toward the same managerial 
objective. The ease with which systems may be described 
in multiattribute terms suggests that the use of this 
approach may facilitate the explicit consideration of 
alternative designs. 



King, William R. and Jaime I. Rodriguez 
MIS Quarterly , pp. 43-51. 

September 1978. 

Most evaluations of information systems are provided only 
in efficiency-oriented terms on a post hoc basis by system 
users . 

Attitudes and value perceptions are an important and often 
neglected aspect of MIS evaluation. 
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The MIS evaluation process is thus a dynamic one in which 
each systematic evaluation effort can lead both to better 
information systems and to improved evaluation 
methodologies . 



Larcker, David F. and Parker V. Lessig 

Perceived Usefulness of Information: A Psychometric 

Examination 

Decision Sciences , vol.ll, pp. 121-134. 

1980. 

Perceived importance will refer to the quality that causes 
a particular information set to acquire relevance to the 
decision maker. If the information items are a necessary 
input for tasks accomplishment, then this quality will 
tend to increase the perceived usefulness of the set. 

Perceived usableness will refer to the information quality 
that allows a decision maker to utilize the set as an 
input for problem solution. If the information is clearly 
presented and readable or can be easily transformed, then 
this quality will tend to increase perceived usefulness. 

A number of instruments have been employed to "measure" 
the perceived usefulness of information for decision 
making, but these techniques lack any data on the 
instrument reliability and validity. Thus in general 
sense, the previous techniques are inadequate and lead to 
a lack of comparability in research results. 



Lincoln T., 

Do Computer Systems Really Pay-off? 

Information and Management , vol.ll, no.l, pp. 25-34. 
August 1986. 

Despite the wide use of cost-benefit forecasting to 
justify proposed system investment, executives remain 
skeptical of the level of benefits actually achieved. 

Such a situation reinforces suspicion in executive's minds 
that benefits from computer systems are speculative and 
probably marginal. 

Post-audits of established systems, however do not have 
such an obvious basis for comparison and it is essential 
to decide what the comparitor will be before undertaking 
a study. 
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SESAME studies however are concerned with overall cost- 
effectiveness and it is usually possible for system 
boundaries to be defined which are meaningful to 
management and allow fairly straight-forward cost 
allocation. The objective is to produced a cost profile 
which is accepted by senior executives as realistic and 
comprehensive . 

Increasingly we see a reluctance to invest further until 
concrete evidence is available that existing system 
investments have produced reasonable financial returns. 

User benefits are frequently declared as "intangible" and 
rarely post-audited. 

All too frequently there is a lack of financial discipline 
applied to the system development process: it is often 
unclear who is accountable for benefit achievement. 



Lucas Jr., Henry C., 

System Quality, User Reactions and The Use of Information 
Systems . 

Management Informatics , vol.3, no. 4, pp. 207-212. 

1974 . 

The findings of this study suggest that before undertaking 
new systems, the quality of existing systems as rated by 
users should be satisfactory. It may be necessary to 
enhance or redesign existing systems before undertaking 
the development of new ones. 

A user's past experience is his best predictor of the 
benefits of a new system, and successfully functioning 
systems are prerequisite to obtaining user cooperation in 
system design. 

Though causality has not been demonstrated, a priory 
arguments and the data support the goals of developing 
favorable user attitudes and high quality systems to 
encourage the use of information systems. 



Lucas Jr., Henry C., 

Performance and The Use of an Information System 
Management Science , vol.21, no. 8, pp. 908-919. 

April 1975. 

Information can be used both to find and solve problems, 
though most current information systems seem to be 
oriented toward problem finding. Possibly problem finding 
data can be provided under exception conditions specified 
by users. 
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One of the most important implications of the model and 
results is that different personal, situational and 
decision style variables appear to affect the use of 
systems. These findings argue for more flexible systems 
to support different user' needs. 

If attitudes are highly negative, on the other hand, users 
will minimize the contribution of the system and question 
the validity of output reports. Attitudes and perceptions 
of computer systems should be influenced in turn by the 
quality of the system as perceived by users. 



Lucas Jr., Henry C. 

Application and Implementation (Unsuccessful 
Implementation: The Case of A Computer-Based Order Entry 

System) . 

Decision Sciences , vol.21, no. 8, pp. 908-919. 

April 1975. 

Dependable variables have usually focused on some measure 
of the level of model or computer system usage. Where use 
is inappropriate, researchers have employed measures of 
satisfaction and/or favorable user attitudes as indicates 
of successful implementation. 



Lucas Jr, Henry C., 

The Use of Accounting Information System, Action and 

Organizational Performance 

The Accounting Review , pp. 735-746. 

October 1975 

In a field study it is difficult to obtain good measures 
for operationalized variables and to control for 
confounding variables. The lack of extremely strong 
results can be partially attributed to errors in measuring 
the variables which were included in the research. 

Laboratory experiments make it possible to control better 
for confounding variables, particularly in measuring 
performance. In addition to interviews and 

questionnaires, the use of protocols may be necessary to 
develop an understanding of how information is integrated 
with the decision-making process. 
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Mansour, Ali H. and Hugh J. Watson 

The Determinants of Computer Based Information System 
Performance 

Academy of Management Journal , vol.23, no. 3, pp. 521-533. 
September 1980. 

The effectiveness of the relationship between system 
personal and others in the organization have an impact on 
system performance. (They looked at from Behavioral 
Variable) . 



Mason, Richard 0., 

Measuring Information Output: A Communication Systems 

Approach 

Information and Management , vol.l, no. 5, pp. 219-234. 
October 1978. 

Notes, that measures of behavior require direct 
observation. The most frequently used measures of change 
in behavior are measures of ability to perform problem 
solving tasks and tests of skills, although most measures 
of change in behavior are unique to each situation and are 
difficult to identify in a complex situation. 

Productivity may be measured as a ratio of useful output 
to the total number of executions of a unit operation. 

Our concerns for efficiency require that the output of 
these systems be related to the input resources consumed. 
Only then can managerial judgments be made as to the 
appropriateness of the allocations. 

There is an important class of systems in our society, for 
which output measures are more difficult to define. These 
are systems whose primary output is information. 



Matlin, Gerald. 

What is The Value of Investment in Information System 
MIS Quarterly , vol.3, no. 3, pp.5-34. 

September 1979. 

Difficulty in finding a value is not an appropriate reason 
for avoiding the subject. 

Are they getting what they paid for in MIS? 

What is the value of the company's investment in 
Information Systems? 
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Managers intuitively know that the most of the benefits 
of IS investments are intangible, difficult to measure, 
and difficult to relate to profit result. 

If the expenditure for the system investment is approved 
by appropriate levels of management and if other valuation 
methods are not appropriate to identify all benefits, then 
the value of the IS investment is equal to the standard 
of ROI in the company times the cost of the IS investment. 



Miller, Jonathan and Brenda A. Doyle 

Measuring the Effectiveness of Computer-Based Information 

System in the Financial Sector 

MIS Quarterly , vol.ll, no.l, pp. 107-124. 

March 1987. 

It was found that performance varies widely between firms 
and that the better performers were those where the level 
of performance matched the perceived importance of the 
particular aspect of IS. 

IS effectiveness here is, at least in part, a function of 
the relationship between perceived importance and 
performance on individual information systems attributes. 



Money, Arthur., David Tromp and Trevor Wegner 

The Quantification of Decision Support Benefits Within the 
Context of Value Analysis 
MIS Quarterly , pp. 223-236. 

June 1988. 

A DSS that is perceived to have value by the potential 
users is more likely to be adopted and implemented than 
one for which no significant value is perceived. 

The technique used, namely conjoint measurement, is 
considered a robust procedure for converting subjective 
judgments about the relative importance of intangible 
benefits into numeric scores for further analysis. 
Traditional cost-benefit approaches to evaluating the 
effectiveness of decision support systems are generally 
regarded as inadequate. More recent approaches account 
for the intangible benefit analysis, but still attempt to 
express benefits in terms of costs. 

Consequently, a more rigorous interpretation of the 
perceived benefits of a DSS, which are a measure of value 
of the proposed DSS, should greatly facilitate the 
decision on whether to proceed with the next stage of 
development or acquisition. 
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Neumann, Seev and Eli Segev 

A Case Study of User Evaluation of Information 
Characteristics for Systems Improvement 
Information and Management , vol.2, pp. 271-278. 

1979. 

The user of the information and his needs are the key for 
the system design. If the system does not fit the user's 
needs, it will not be used. Extent of use serves in this 
case as an indication of the success of the system. 

Improvement of an existing system is aimed at providing 
a better response to user needs, that is, increasing the 
use of the system by increasing user satisfaction from 
existing system, it is necessary to first measure his 
current evaluation of the system, find out those aspects 
of the information which receive low evaluation, and then 
improve them. 

The findings of this study indicate that the users at the 
bank had a holistic perception of their information, and 
that content was a dominant characteristic which spilled 
over other characteristics. 



Poppel, Harvey L. 

Who needs the office of the future? 

Harvard Business Review , vol.54, no. 5, pp. 146-155. 
November-December 1982. 

Almost everyone could benefit from the new technology, 
according to this study of white-collar productivity. 

Despite the allure, however, businesses are spending less 
than a nickel for such technology of every dollar of 
knowledge-worker salaries and internal support costs. 

The fact is that most decision makers are skeptical about 
what managerial work stations, personal computers, video- 
conference rooms, and the other, newer icons of office 
automation can do for their businesses. These executives, 
disenchanted by their pervious exposure to ill-conceived 
forays into management information systems and word 
processing, doubt that knowledge workers will embrace the 
new technology, and they lack confidence that their 
organizations can channel and measure the intended 
benefits. In addition, members of the computer illiterate 
majority are worried about whether, when, and how they 
themselves will deal with these new electronic tools. 
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Raghunatan, TS . and W. R. King 

The Impact of Information Systems Planning on the 
Organization 

Omega , vol.16, no. 2, pp. 85-93. 

September 1987. 

"...Attempts to relate planning to overall business 
performance, measured in terms such as return-on- 
investment (ROI), are not fruitful, primarily because even 
such an important activity as strategic business planning 
is only one of many factors that influence ROI." 

These studies, together with the limited ones done in an 
IS planning context, suggest that IS planning may be most 
fruitfully evaluated using a surrogate measure of its 
organizational impact and measures of the specific 
elements that make up the planning activity. 



Rockart, John F. 

The Changing Role of the Information Systems Executive: A 

Critical Success Factors Perspective 

Sloan Management Review , vol.24, no.l, pp.3-14. 

Fall 1982. 

Three of the four critical success factors (communication, 
I/S human resources, and repositioning) directly reflect 
this evolving staff-oriented role definition. Operating 
responsibility for local I/S units is increasingly being 
transferred to local line managements. The top I/S 
executive today is a "thinker, planner, and coordinator" 
rather than a direct "implementor and doer." 



Ruby, Daniel and Richard L. Zeller 

Factors affecting the Success and Failure of an Information 
System for Product Quality 
Interfaces , vol.8, no. 3, pp. 70-75. 

February 1978. 

As a research strategy, the comparative case study 
reported here strikes a balance between large sample 
survey techniques and one-shot case studies. It neither 
possesses the severe limitations of these forms of 
research nor makes full use of their advantages. 

Our case indicates the importance of several factors in 
MIS implementation. 

1. At the individual level, certain attitudes were 
found to be more important than others . 

2. The organizational factors of complexity, formality, 
and centralization also affect implementation. 

3. Lack of involvement by system developers is not 
sufficient to ensure failure if the vital function 
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of explaining the system to ultimate users assumed 
by some other knowledgeable person in the group. 

4 . Strong management support is instrumental to system 
adoption . 



Ruby, Daniel. 

User Attitudes and Management Information System Use 
Academy of management Journal , vol.22, no. 3, pp. 527-538. 
September 1979. 

Overall, these results support the established notion that 
user attitudes (or perceptions) are significant correlates 
of system use. Attitudes are less powerful in predicting 
subjective assessments of perceived worth, although the 
relationships are significant. The finding that attitudes 
are more strongly related to actual use than they are to 
measures of perceived worth has important implications. 

User concerns are critical to success of MIS, particularly 
concerns about the impact of MIS on individual 

performance. Obviously, these concerns should be 
addressed during implementation, but in some cases it may 
be too late by then. 

A system that does not help people perform their jobs is 
not likely to be received favorably in spite of careful 
implementation efforts. A system that reduces rewards for 
users is likely to meet with disaster. A logical and 
often recommended approach to systems design is to involve 
users in the design effort. Systems designers and 
implementation teams would be well advised to find some 
means of addressing these concerns during MIS development. 



Snitkin, Sidney R. and William R. King 

Determinants of the Effectiveness of Personal Decision 
Support Systems 

Information and Management , vol.10, no. 2, pp. 83-89. 
January 1986. 

The overall effectiveness of the system was assessed as 
is typical in such studies, using a perceptual measure. 
Overall, this study presents an interesting perspective 
on the evolving notion of a "personal DSS . " The evidence 
that is analyzed here suggest that the pragmatic notion 
of personal DSSs has meaning and that some determinants 
of the effectiveness of such systems can be identified. 
While such systems do not meet the formal criteria for 
DSSs as they were initially conceived, they are important 
to the everyday functioning of business organizations, and 
as such. 
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Srinivasan, Ananth, . 

Alternative Measures of System Effectiveness: Associations 
and Implications 

MIS Quarterly , vol.9, no. 3, pp. 243-253. 

September 1985. 

While acknowledging the importance of economic analysis 
of MIS value, researchers responds to the shifting 
emphasis from efficiency to user effectiveness by focusing 
either on MIS usage or user perceived effectiveness. 

From a practitioners perspective, it is important to 
understand what exactly is being measured when a system 
effectiveness study is initiated in an organization. 



Sullivan, Robert S. and Stephen C. Secrest 

A Simple Optimization DSS for Productivity Planning at 
Dairyman's Cooperative Creamery Association 
Interfaces , vol.15, no. 5, pp. 46-53. 

September-October 1985. 

The milk flow analysis program, used for daily production 
planning and inventory forecasting, relieves plant 
supervisors of about four hours a day of overwhelming hand 
calculations . 

In addition, the system has improved interaction between 
all levels of management. 

The managers of milk processing plants in the United 
States rarely use computer-based production planning and 
decision support system. 

The benefits from using MFAP are obvious but somewhat 
difficult to quantify (Then they aren't obvious). These 
benefits are the result of improved scheduling on the 
evaporator, the bottle-neck operation. MFAP has increased 
daily plant throughput by an estimated 150,000 pounds per 
day . 



Swanson, Burton E., 

Management Information Systems: Appreciation and 

Involvement . 

Management Sciences , vol.21, no. 2, pp. 178-188. 

October 1974. 

It seems reasonable to conclude that MIS appreciation 
coproduces (and is coproduced by) MIS involvement. 
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APPENDIX F 



TABLES OF FINDINGS 
Tabltt F-l: R*s«arch Design Typ** 



1 23 Campbell 

Pre Comp Random & Stanley 



Study Year Test Grp Assign Design Type 



1 


Colton 


1973 








One shot case study 


2 


Lucas 


1973 




X 




Static group comparison 


3 


Swanson 


1974 








One shot case study 


4 


Gallagher 


1974 








One shot case study 


5 


Lucas 


1975 








One shot case study 


6 


Lucas 


1975 








One shot case study 


7 


Hall & Lincoln 


1976 




X 




Static group comparison 


8 


Barkin & Dickson 


1977 




X 




Static group comparison 


9 


Edstrom 


1977 








One shot case study 


10 


King & Rodriguez 


1978 


X 


X 


X 


True experiment 


11 


Robey & Zeller 


1978 






X 


Static group comparison 


12 


Mason 


1978 








One shot case study 


13 


Lucas 


1978 


X 


X 


X 


True experiment 


14 


Matlin 


1979 








One shot case study 


15 


Neumann & Segev 


1979 








One shot case study 


16 


Robey 


1979 








One shot case study 


17 


Mansour & Watson 


1980 








One shot case study 


18 


Ball 


1980 




X 




One group pretest /posttest 


19 


Larcker & Lessig 


1980 








One shot case study 


20 


Edelman 


1981 






X 


Static group comparison 


21 


Altmeyer & Bozeman 


1981 








One shot case study 


22 


Rockart 


1982 








One shot case study 


23 


Epstein & King 


1982 






X 


Static group comparison 


24 


Feurst & Cheney 


1982 








One shot case study 


25 


Poppel 


1982 








One shot case study 


26 


Bailey & Pearson 


1983 








One shot case study 


27 


Cron & Sobol 


1983 








One shot case study 


28 


King & Epstein 


1983 








One shot case study 


29 


Bruwer 


1984 








One shot case study 


30 


Decker & Plumlee 


1985 








One shot case study 


31 


Englander & Englander 


1985 


X 






Time series 


32 


Srinivasan 


1985 








One shot case study 


33 


Sullivan & Secrest 


1985 




X 




One group pretest /posttest 


34 


Hogue & Watson 


1985 








One shot case study 


35 


Lincoln 


1986 






X 


Static group comparison 


36 


Franz & Robey 


1986 








One shot case study 


37 


Kapsales 


1986 








One shot case study 


38 


Snitkin & King 


1986 








One shot case study 


39 


Aldag & Power 


1986 




X 


X 


Post test only with control group 


40 


Miller & Doyle 


1987 








One shot case study 


41 


Durand et al 


1987 




X 


X 


Posttest only with control group 


42 


Raghunathan 


1988 








One shot case study 


43 


Baroudi & Orlikowski 


1988 








One shot case study 


44 


Money et al 


1988 








One shot case study 


45 


Gallupe et al 


1988 




X 




Static group comparison 
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Table F-2 : Threats to Internal Validity 



Sal 



STUDY 


YEAR 


His 


Mat 


Tst 


Ins 


Rea 


Sel 


Mor 


Mat 


1 


Colton 


1973 


-1 


-1 








-1 


-1 




2 


Lucas 


1974 


+1 


7 


+1 


+1 


-1 


-1 


-1 




3 


Swanson 


1974 


-1 


-1 










-1 


-1 


4 


Gallagher 


1974 


-1 


-1 










-1 


-1 


5 


Lucas 


1975 


-1 


-1 










-1 


-1 


6 


Lucas 


1975 


-1 


-1 










-1 


-1 


7 


Hall £ Lincoln 


1976 


+ 1 


7 


+1 


+ 1 


+ 1 


-1 


-1 


-1 


8 


Barkin £ Dickson 


1977 


+ 1 


7 


+ 1 


+ 1 


-1 


-1 


-1 




9 


Edstrom 


1977 


-1 


-1 










-1 


-1 


10 


King £ Rodriguez 


1978 


+1 


+ 1 


+1 


+1 


+1 


+1 


+1 


+ 1 


11 


Robey £ Zeller 


1978 


+ 1 


7 


+1 


+1 


-1 




-1 




12 


Mason 


1978 


-1 


-1 










-1 


-1 


13 


Lucas 


1978 


+ 1 


+ 1 


+1 


+ 1 


+1 


+1 


+1 




14 


Matlin 


1979 


-1 


-1 










-1 


-1 


15 


Neumann £ Segev 


1979 


-1 


-1 










-1 


-1 


16 


Robey 


1979 


-1 


-1 










-1 


-1 


17 


Mansour £ Watson 


1980 


-1 


-1 










-1 


-1 


18 


Ball 


1980 


-1 


-1 


-1 


-1 


7 


+1 


+1 


-1 


19 


Larcker £ Lessig 


1980 


-1 


-1 










-1 


-1 


20 


Edelman 


1981 


+ 1 


7 


+1 


+1 


+ 1 


-1 


-1 


-1 


21 Altmeyer £ Bozeman 


1981 


-1 


-1 










-1 


-1 


22 


Rockart 


1982 


-1 


-1 










-1 


-1 


23 


Epstein £ King 


1982 


+1 


7 


+ 1 


+1 


+1 


-1 


-1 


-1 


24 


Feurst £ Cheney 


1982 


-1 


-1 










-1 


-1 


25 


Poppel 


1982 


-1 


-1 










-1 


-1 


26 


Bailey £ Pearson 


1983 


-1 


-1 










-1 


-1 


27 


Cron £ Sobol 


1983 


-1 


-1 










-1 


-1 


28 


King £ Epstein 


1983 


-1 


-1 










-1 


-1 


29 


Bruwer 


1984 


-1 


-1 










-1 


-1 


30 


Decker & Plumlee 


1985 


-1 


-1 










-1 


-1 


31 


Englander £ Englander 


1985 


-1 


+1 


+ 1 


7 


+ 1 


+1 


+1 


+1 


32 


Srinivasan 


1985 


-1 


-1 










-1 


-1 


33 


Sullivan £ Secrest 


1985 


-1 


-1 


-1 


-1 


7 


+ 1 


+1 


-1 


34 


Hogue £ Watson 


1985 


-1 


-1 










-1 


-1 


35 


Lincoln 


1986 


+1 


7 


+ 1 


+1 


+ 1 


-1 


-1 


-1 


36 


Franz £ Robey 


1986 


-1 


-1 










-1 


-1 


37 


Kapsales 


1986 


-1 


-1 










-1 


-1 


38 


Snitkin £ King 


1986 


-1 


-1 










-1 


-1 


3 9 Aldag £ Power 


1986 


+1 


+ 1 


+ 1 


+ 1 


+ 1 


+ 1 


+ 1 


+1 


40 


Miller £ Doyle 


1987 


-1 


-1 










-1 


-1 


41 


Durand et al 


1987 


+ 1 


+1 


+ 1 


+ 1 


+ 1 


+1 


+ 1 


+1 


42 


Raghunathan 


1988 


-1 


-1 










-1 


-1 


43 


Baroudi £ Orlikowski 


1988 


-1 


-1 










-1 


-1 


44 


Money et al 


1988 


-1 


-1 










-1 


-1 


45 


Gallupe et al 


1988 


+ 1 


7 


+1 


+ 1 


+ 1 


-1 


-1 


-1 
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Table F-3: Threats bo External Validity 



Interactions : 
Test Selct 



Study 


Year 


& X 


& X 


1 


Colton 


1973 




-1 


2 


Lucas 


1974 




-1 


3 


Swanson 


1974 




-1 


4 


Gallagher 


1974 




-1 


5 


Lucas 


1975 




-1 


6 


Lucas 


1975 




-1 


7 


Hall & Lincoln 


1976 




-1 


8 


Barkin & Dickson 


1977 




-1 


9 


Edstrom 


1977 




-1 


10 


King & Rodriguez 


1978 


-1 


7 


11 


Robey & Zeller 


1978 




-1 


12 


Mason 


1978 




-1 


13 


Lucas 


1978 


-1 


7 


14 


Matlin 


1979 




-1 


15 


Neumann & Segev 


1979 




-1 


16 Robey 


1979 




-1 


17 


Mansour & Watson 


1980 




-1 


18 


Ball 


1980 


-1 


-1 


19 


Larcker & Lessig 


1980 




-1 


20 


Edelman 


1981 




-1 


21 


Altmeyer & Bozeman 


1981 




-1 


22 


Rock art 


1982 




-1 


23 


Epstein & King 


1982 




-1 


24 


Feurst & Cheney 


1982 




-1 


25 


Poppel 


1982 




-1 


26 


Bailey & Pearson 


1983 




-1 


27 


Cron & Sobol 


1983 




-1 


28 


King & Epstein 


1983 




-1 


29 


Bruwer 


1984 




-1 


30 


Decker & Plumlee 


1985 




-1 


31 


Englander & Englander 


1985 


-1 


7 


32 


Srinivasan 


1985 




-1 


33 


Sullivan & Secrest 


1985 


-1 


-1 


34 


Hogue & Watson 


1985 




-1 


35 


Lincoln 


1986 




-1 


36 


Franz & Robey 


1986 




-1 


37 


Kapsales 


1986 




-1 


38 


Snitkin & King 


1986 




-1 


3 9 Aldag & Power 


1986 


+1 


7 


40 


Miller & Doyle 


1987 




-1 


41 


Durand et al 


1987 


+1 


7 


42 


Raghunathan 


1988 




-1 


43 


Baroudi & Orlikowski 


1988 




-1 


44 


Money et al 


1988 




-1 


45 


Gallupe et al 


1988 




-1 



Reactv 

Arrq 



7 



7 



7 



7 

7 



7 

7 



Multiple 

X 

Inttrf 



129 



Table F-4: Handling of Time Dimension 



STUDY 



1 Colton 

2 Lucas 

3 Swanson 

4 Gallagher 

5 Lucas 

6 Lucas 

7 Hall & Lincoln 

8 Barkin & Dickson 

9 Edstrom 

10 King & Rodriguez 

11 Robey & Zeller 

12 Mason 

13 Lucas 

14 Matlin 

15 Neumann & Segev 

16 Robey 

17 Mansour & Watson 

18 Ball 

19 Larcker & Lessig 

20 Edelman 

21 Altmeyer & Bozeman 

22 Rockart 

23 Epstein & King 

24 Feurst & Cheney 

25 Poppel 

26 Bailey St Pearson 

27 Cron St Sobol 

28 King St Epstein 

29 Bruwer 

30 Decker St Plumlee 

31 Englander St Englander 

32 Srinivasan 

33 Sullivan St Secrest 

34 Hogue St Watson 

35 Lincoln 

3 6 Franz St Robey 

37 Kapsales 

38 Snitkin & King 

39 Aldag & Power 

40 Miller St Doyle 

41 Durand et al 

42 Raghunathan 

43 Baroudi & Orlikowski 

44 Money et al 

45 Gallupe et al 



YEAR 


Handled 

Tim* 

Dimension 


1973 


No 


1974 


No 


1974 


No 


1974 


No 


1975 


No 


1975 


No 


1976 


No 


1977 


No 


1977 


No 


1978 


Yes 


1978 


No 


1978 


No 


1978 


Yes 


1979 


No 


1979 


No 


1979 


No 


1980 


No 


1980 


Yes 


1980 


No 


1981 


No 


1981 


No 


1982 


No 


1982 


No 


1982 


No 


1982 


No 


1983 


No 


1983 


No 


1983 


No 


1984 


No 


1985 


No 


1985 


Yes 


1985 


No 


1985 


Yes 


1985 


No 


1986 


No 


1986 


No 


1986 


No 


1986 


No 


1986 


No 


1987 


No 


1987 


No 


1988 


No 


1988 


No 


1988 


No 


1988 


No 
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T&bl* F-5 : Data Collact ion Tachniquaa 



Study 


Year 


1st 

Method 

Used 


2nd 

Method 

Used 


1 


Colton 


1973 


Mailed Questionnaire 


Interview Survey 


2 


Lucas 


1974 


Questionnaire 




3 


Swanson 


1974 


Archival 


Questionnaire 


4 


Gallagher 


1974 


Questionnaire 


Smantic Differential 


5 


Lucas 


1975 


Questionnaire 




6 


Lucas 


1975 


Mailed Questionnaire 


Archival 


7 


Hall & Lincoln 


1976 


Interview Survey 




8 


Barkin & Dickson 


1977 


Questionnaire 


Simulation 


9 


Edstrom 


1977 


Interview Survey 




10 


King & Rodriguez 


1978 


Questionnaire 




11 


Robey & Seller 


1978 


Questionnaire 




12 


Mason 


1978 


Quest ionnaire ? 




13 


Lucas 


1978 


Mailed Questionnaire 


Questionnaire 


14 


Matlin 


1979 


Archival 




15 


Neumann & Segev 


1979 


Questionnaire 




16 


Robey 


1979 


Archival 


Mailed Questionnaire 


17 


Mansour & Watson 


1980 


Mailed Questionnaire 




18 


Ball 


I960 


Questionnaire 


Archival 


19 


Larcker & Lessig 


1980 


Questionnaire 




20 


Edelman 


1981 


Archival 




21 


Altmeyer & Bozeman 


1981 


Phone Interview 


Mailed Questionnaire 


22 


Rockart 


1982 


Interview Survey 




23 


Epstein & King 


1982 


Questionnaire 




24 


Feurst & Cheney 


1982 


Mailed Questionnaire 


Mailed Questionnaire 


25 


Poppel 


1982 


Diary 


Questionnaire 


26 


Bailey & Pearson 


1983 


Interview Survey 


Mailed Questionnaire 


27 


Cron & Sobol 


1983 


Questionnaire 




28 


King & Epstein 


1983 


Questionnaire 




29 


Bruwer 


1984 


Questionnaire 




30 


Decker & Plumlee 


1985 


Archival 




31 


Englander & Englander 


1985 


Archival 




32 


Srinivasan 


1985 


Mailed Questionnaire 




33 


Sullivan & Secrest 


1985 


Archival 




34 


Hogue & Watson 


1985 


Interview Survey 


Questionnaire 


35 


Lincoln 


1986 


Questionnaire 




36 


Franz & Robey 


1986 


Questionnaire 




37 


Kapsales 


1986 


Archival 




38 


Snitkin & King 


1986 


Mailed Questionnaire 




39 Aldag & Power 


1986 


Laboratory Experiment 


Questionnaire 


40 


Miller & Doyle 


1987 


Mailed Questionnaire 




41 


Durand et al 


1987 


Laboratory Experiment 




42 


Raghunathan 


1988 


Mailed Questionnaire 




43 


Baroudi & Orlikowski 


1988 


Mailed Questionnaire 




44 


Money et al 


1988 


Questionnaire 




45 


Gallupe et al 


1988 


Panel 


Questionnaire 
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Tabl* F-€: Definitions of Effectiveness 



Study Year 



1 Colton 1973 

2 Lucas 1974 

3 Swanson 1974 

4 Gallaghar 1974 

5 Lucas 1975 

6 Lucas 1975 

7 Hall & Lincoln 1976 

8 Barkin 6 Dickson 1977 

9 Edstrom 1977 

10 King & Rodriguez 1978 

11 Robay & Zeller 1978 

12 Mason 1978 

13 Lucas 1978 

14 Matlin 1979 

15 Naumann & Sagav 1979 

16 Robay 1979 

17 Mansour & Watson 1980 

18 Ball 1980 

19 Larckar 6 Lassig 1980 

20 Edelman 1981 

21 Altmeyar & Bozeman 1981 

22 Rockart 1982 

23 Epstein 6 King 1982 

24 Feurst & Cheney 1982 

25 Poppel 1982 

26 Bailey & Pearson 1983 

27 Cron & Sobol 1983 

28 King & Epstein 1983 

29 Bruwer 1984 

30 Decker & Plumlee 1985 

31 Englander & Englander 1985 

32 Srinivasan 1985 

33 Sullivan & Secrest 1985 

34 Hogue & Watson 1985 

35 Lincoln 1986 

36 Franz & Robey 1986 

37 Kapsales 1986 

38 Snitkin & King 1986 

39 Aldag & Power 1986 

40 Miller & Doyle 1987 

41 Durand at al 1987 

42 Raghunathan 1988 

43 Baroudi & Orlikowski 1988 

44 Money at al 1988 

45 Gallupe at al 1988 



Definition of System Payoff 

System usage 

Attitude of users toward system 
System usage 

Dollar value information 
System usage 
System usage 
Cost savings 
Information usage 
User influence 

Attitude of users toward system 
Attitude of users toward system 
Problem solving ability 
System usage 
Value/cost ratio 

Attitude of users toward system 

System value 

Performance factors 

Attitude of users toward system 

Information usefulness 

Cost avoidance 

Attitude of users toward system 
Job factors 

Attitude of users toward system 

System usage 

Office automation 

Attitude of users toward system 

Return on investment 

Information value 

Attitude of users toward system 

Not clear 

Cost savings 

Attitude of users toward system 
Profit 

System usage 
Cost benefit 
System usefulness 
Cost avoidance 

Attitude of users toward system 
Decision quality 

Attitude of users toward system 
Grammar 

Implementation of plan 
Attitude of users toward system 
Attitude of users toward system 
Decision quality 
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Table F-7 : Reports of Change in System Effectiveness 



Study 



Reported 
Year Change 



1 


Colton 


1973 


Not 


reported 


2 


Luca a 


1974 


Not 


reported 


3 


Swanson 


1974 


Not 


reported 


4 


Gallagher 


1974 


Not 


reported 


5 


Lucas 


1975 


Not 


reported 


6 


Lucas 


1975 


Not 


reported 


7 


Hall & Lincoln 


1976 


25% 




8 


Barkin & Dickson 


1977 


Not 


reported 


9 


Edst rom 


1977 


Not 


reported 


10 


King & Rodriguez 


1978 


Not 


reported 


11 


Robey & Zeller 


1978 


Not 


reported 


12 


Mason 


1978 


? 




13 


Lucas 


1978 


Not 


reported 


14 


Matlin 


1979 


1.1 


to 3 . 3 


15 


Neumann & Segev 


1979 


Not 


reported 


16 


Robey 


1979 


Not 


reported 


17 


Mansour & Watson 


1980 


Not 


reported 


18 


Ball 


1980 


Not 


reported 


19 


Larcker & Lessig 


1980 


Not 


reported 


20 


Edelman 


1981 


15 i 


to 20% 


21 


Altmeyer & Bozeman 


1981 


Not 


reported 


22 


Rockart 


1982 


Not 


reported 


23 


Epstein & King 


1982 


Not 


reported 


24 


Feurst & Cheney 


1982 


Not 


reported 


25 


Poppel 


1982 


Not 


reported 


26 


Bailey & Pearson 


1983 


Not 


reported 


27 


Cron & Sobol 


1983 


? 




28 


King & Epstein 


1983 


Not 


reported 


29 


Bruwer 


1984 


Not 


reported 


30 


Decker & Plumlee 


1985 


? 




31 


Englander & Englander 


1985 


4 9% 




32 


Srinivasan 


1985 


Not 


reported 


33 


Sullivan & Secrest 


1985 


$100,000 


34 


Hogue & Watson 


1985 


Not 


reported 


35 


Lincoln 


1986 


13- 


400% 


36 


Franz & Robey 


1986 


Not 


reported 


37 


Kapsales 


1986 


102% 


38 


Snitkin & King 


1986 


Not 


reported 


39 Aldag & Power 


1986 


No 


improvement following implementation 


40 


Miller & Doyle 


1987 


Not 


reported 


41 


Durand et al 


1987 


Not 


reported 


42 


Raghunathan 


1988 


Not 


reported 


43 


Baroudi & Orlikowski 


1988 


Not 


applicable 


44 


Money et al 


1988 


Not 


reported 


45 


Gallupe et al 


1988 


Group decision support system supported better 
quality of decisions 
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